On a recent project I ran into a unique problem regarding replication of public folder information in an Exchange 2003 organization. The problem was recently discovered after the migration of an acquisition into the existing organization when users noticed that Free/Busy information for mailboxes was not displaying between mailboxes in the existing offices and the new office. After an exhaustive search and a few dead-ends I was able to finally resolve the issue, but the lack of information I found online made this a good topic for review.
To setup the scenario, we have a multiple-server Exchange 2003 organization with a single routing group and single administrative group. No public folders were using replication, but a few system folders were configured to replicate between 4 mailbox servers. A new company was acquired and their current Exchange 2000 information was migrated (via Quest Migration Manager) onto a new Exchange 2003 mailbox server which was located in a new administrative group.
After the migration was completed it was reported that users on mailbox servers in Administrative Group 1 (AG1) could not view Free/Busy information for any users in Administrative Group 2 (AG2), and vice versa. All other mail flow worked correctly and all Exchange servers were allowed to communicate with each other across all ports between any firewalls.
The first thing I looked at was the replication settings for the SCHEDULE+ FREE BUSY system folder, as that is the where all mailbox free/busy information is stored in Exchange 2003. On Exchange servers in the first administrative group I only saw a single subfolder for that administrative group, and the new server only contained a subfolder for the second administrative group. I quickly compared the public folder hierarchy between servers in each administrative group and saw that they did not match. It turned out that users in each group saw only their ‘local’ folders, which from a user standpoint caused no issues. Corporate users saw no change in the folder hierarchy and migrated users from the new company still saw only their migrated folder structure as it appeared before. But now that existing users were trying to schedule meetings with attendees from the newly acquired company did the underlying issue of a segmented public folder hierarchy become apparent.
To begin troubleshooting public folder replication problems, logging needs to be enabled so that individual messages can be tracked. This Microsoft TechNet article (KB842273) explains how to enable logging on the correct sources and which application event logs correlate to specific events. After some message tracking it was clear that outbound replication messages from a server in AG1 was reaching all servers in the same group, but not the new server in AG2. I also verified that replication messages outbound from the new server were not reaching any server in the AG1. After verifying that firewalls between both sites were not causing any problems I kept on searching for the root cause.
Coincidentally this related MSExchange.org tutorial mentions that one possible cause might be missing a SMTP address on a Public Store, which would prevent the server from sending and receiving SMTP-based PF replication messages. Sure enough, the proxyAddresses attribute on the new server’s Public Folder Store was completely blank. This was the problem.
My first concern was that although I could manually stamp this attribute and probably resolve the replication issues quickly, what caused this mis-configuration in the first place? The Enterprise Recipient Update Service is responsible for stamping that address on the PF store in the first place, so that is where I turned my attention. Using TechNet article 822794 is a guide I ran through a couple cycles of both the Update and Rebuild All commands, but after verifying (via USN) that the targeted object was processed the attributes continued to remain blank. I checked the gatewayProxy attribute on the Enterprise RUS itself, to see if the ‘queue’ was jammed up (as described in TechNet article 821743) but it was clear.
At this point I shifted focus back to getting the replication issue resolved and would save the root cause troubleshooting for later. I manually stamped the proxyAddresses attribute with SMTP and X400 addresses, using the formats SMTP:SERVERNAME-IS@domain.com and duplicated the X400 address from another server, updating the Administrative Group name in the path. After some time passed I still didn’t see any replication messages flowing in or out of that server. I finally located Microsoft documentation related to this specific issue, under the Storage section of the Best Practice Analyzer articles, entitled Public Folder store does not have an email-address. I found that both the mail and textEncodedORAddress attributes where also blank, so I populated them and double-checked ALL mail-related attributes on the same Public Folder Store object using ADSIedit.
Finally, within an hour, I was finding multiple replication messages in the tracking logs being delivered successfully to and from the new server. Yet, strangely, I still didn’t see the SCHEDULE+ child folder for the other Administrative Groups appearing on servers in either group. I manually triggered Public Folder Hierarchy replication and at last, I began to see Hierarchy, and Folder Content Backfill Response messages:
By this point the entire hierarchy trued up between all servers and both folders and content began to populate between all Exchange servers.
With the immediate problem of SMTP mail flow into the public store resolved, I set off to discover the root cause of the issue. I knew that the Enterprise RUS was not stamping those attributes like it was supposed to, but nothing in the error logs was helping. Shortly after, the client deployed another Exchange mailbox server and noticed the same problem with missing mail attributes on the public folder store. When they cranked up diagnostic logging on the server holding the Enterprise RUS service they noticed something that we hadn’t seen when troubleshooting the issue with the previous server. There were permissions-related errors (IDs 8317 and 8270) appearing under the ExchangeAL service in the LDAP Operations category, basically reporting that the server running the ERUS was trying to stamp those objects, but was unable. Scenario 3 in TechNet article 254030 matched the specific errors we were seeing.
The client immediately linked this to an undocumented customization of their environment where the default security permissions had been altered way back when Exchange 2003 was originally deployed. As a practice, whenever a new server was built they manually added the other Exchange server objects to the new server’s Security tab, granting Full Control rights. This configuration was unknown to the migration team and has since been added to the server deployment documentation as a required build step.