PrepareAD fails running the Exchange Server 2010 setup

The Problem
We got into a problem recently at a customer site when running the Exchange server 2010, setup.com /prepareAD switch in an Exchange Server 2003 environment. The command failed and exited with the problem pointing to the “Default Global Address List”, according to the error message in the exchange setup log:

[ERROR] Active Directory operation failed on dc1.example.internal. The object 'CN=Default Global Address List,CN=All Global Address Lists,CN=Address Lists Container,CN=ExampleOrg,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=example,DC=internal' already exists.

Research
We searched for the error message on similar cases on the Internet which indicated the need to recreate the “Default Global Address List”.
We didn’t see the missing Default Global Address List, however a new similar empty list called “Default” was there when looking in the Exchange System Manager. Did the customer rename the *Default Global Address List” to just “Default”? The LDAP filter for this “Default” was empty and we couldn’t rename it to “Default Global Address List” or create a new ”Default Global Address List”, it already exists according to the error message.

Found a strange object “Default Global Address List” which was empty of any attributes in the "All Global Address Lists" Container when looking at it with ADSIedit. The only attribute we could see was the Distinguished Name (DN) of “Default Global Address List” in the All Address List Container and another equally strange object “All” with also only a DN and no other attributes populated.

We opened the LDP.exe tool and looked at the object with the same result, couldn’t see any other attributes than the DN of the object, no USN, no WhenChanged, no WhenCreated date, no GUID, nothing but the DN.
Our theory here was the customer had somehow removed the “Default Global Address List” in an unsupported way, and that the remaining object was now some sort of leftover.

Troubleshooting
Tried to delete both "leftover" address lists with ADSIedit, only to get the following error:
---------------------------

Operation failed. Error code: 0x8007200a

The specified directory service attribute or value does not exist

Could it be permission issue, changed the permission for Everyone and Authenticated Users, which had Explicit Read Deny?
Tried to delete both of the "leftover" address lists again with ADSIedit, which resulted in the following error:
---------------------------

Operation failed. Error code: 0x80005008

One or more input parameters are invalid

At least now we knew that we were on the right track and now we could actually see the Default Global Address List in System Manager.

Solution
We removed all the explicitly set permissions for Authenticated Users and Everyone group (we don’t want our users to see all recipients since it’s a hosted solution with many companies and multiple address lists), which also removed any Deny permissions.
We rerun the command d:\setup.com /PrepareAD switch and everything worked just fine this time.

Root Cause
It was the Deny Read permission for the Authenticated Users and Everyone group that blocked the setup program from running the /PrepareAD.

So the root cause to this problem was that the customer in his effort to hide the “Default Global Address List” probably made an administrator mistake and denied everyone the rights to read the “Default Global Address List”. The correct way should have instead been just to remove the Read permission for the Everyone and Authenticated Users Group. This change denied both the administrator and the System the right to read the object. When the Exchange Server 2010 setup run, it couldn’t read and find the “Default Global Address List” during the /PreparedAD phase, it then tries to repair and fix the problem by recreating the “Default Global Address List”, which also fails since the object is still there with its Distinguished Name “ Default Global Address List”.

Following are the interesting parts from the
ExchangeSetup.log confirming the troubleshooting result.

[04-23-2010 10:02:32.0089] [1] Executing 'install-GlobalAddressLists -DomainController $RoleDomainController' failed. The error is: False

[04-23-2010 10:02:32.0089] [2] Launching sub-task '$error.Clear(); install-GlobalAddressLists -DomainController $RoleDomainController'.
[04-23-2010 10:02:32.0105] [2] Active Directory session settings for 'Install-GlobalAddressLists' are: View Entire Forest: 'True', Configuration Domain Controller: 'dc1.example.internal', Preferred Global Catalog: 'dc1.example.internal', Preferred Domain Controllers: '{ dc1.example.internal }'
[04-23-2010 10:02:32.0105] [2] Runspace context: Executing user: Example.internal/Users/Administrator, Executing user organization: , Current organization: , RBAC-enabled: Disabled.
[04-23-2010 10:02:32.0120] [2] Saving object "\Default Global Address List" of type "AddressBookBase" and state "New".

[04-23-2010 10:02:32.0167] [2] [ERROR] Active Directory operation failed on dc1.example.internal. The object 'CN=Default Global Address List,CN=All Global Address Lists,CN=Address Lists Container,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=EXAMPLE,DC=internal' already exists.
[04-23-2010 10:02:32.0167] [2] [ERROR] The object exists.
[04-23-2010 10:02:32.0183] [2] Ending processing.
[04-23-2010 10:02:32.0198] [1] [WARNING] <<< setup failed to execute a task. Dumping all variables.... >>

Prepare a Windows 2003 Active Directory for Exchange Server 2010

                        
Background
I often get the question, what do we need to do to prepare our Active Directory for Exchange 2010?
Well, the short answer is just to pop in the Exchange 2010 DVD and run the GUI setup, which will do it for you.
But what if you want to have more control over the procedure in a phased approach?
In that case you can't run the GUI Setup, you have to do it from the command prompt and use the setup.com program, which this post is about.
The procedure below can actually also be used when introducing new versions of domain controllers as well as preparing Active Directories for Exchange Server 2003 or 2007. However, you need to make sure that the command you run and syntax matches the version you're preparing, for example, you need to use the Adprep/Forestprep when introducing newer versions of domain controllers into the domain and forest.

Environment
In this scenario we have a Windows 2003 Active Directory Forest running in Windows 2000 Forest mode. There are a total of two domain controllers in one Windows 2003 mode domain. One of the DC is running a 32 bit version of Windows Server 2003 SP2 and the other DC is running a 64 bit version of Windows Server 2008 SP2. The Windows 2008 DC is also the FSMO role owner, GC and DNS. The Windows Server 2003 DC is a GC and DNS server.
Exchange 2003 SP2 is installed on four Exchange Server in the environment.

Objective
Is to prepare the Active Directory for Exchange Server 2010 and minimize the risk of service disruption during, under and after the Active Directory preparation. There’s a requirement for a rollback plan hence any non recoverable error should occur. The requirement is also to avoid having both DC's out of service should an error occur due strict SLA and 24x7 production environment.
A successful change mean that no service disruption or noticable change should be seen by any user during the process.

Tests and Execution
As for most successful changes to environments there's a lot of planning and testing involved. The more you plan and test, the more likely it's going to be a successful change. Some of the test below are a bit exessive in an environment with only two DC, however, it's always good to verify and in a large environment it's a must, trust but verify.
The following steps must be performed by a privileged account. The Schema extension and other extensive changes to the directory need to have you logon with an account that's member of the Domain Admins, Enterprise Admins and Schema Admins group.
The procedure below is an iterative process of test, change, verify, test, change, verify, ....
A convenvient way of doing the steps below is to put them in a script file with the correct naming of the domain and DC's of course. This script is test in your lab environment first before used in the production, not run as a whole script automatically, but rather used for cutting and pasting commands into the command prompt. That way you create a structured approach with less risk of misspelling commands and also a faster execution of the process.

1. Run this on the Windows Server 2008 Server

To verify the operating system and ServicePack the following command can be run (Windows 2003 or later version of repadmin run on a Windows Server 2003 or Windows Server 2008/2008R2 server):

To check operating System Level on the domain controller dc1 in the domain example.internal you would type in one line:

repadmin /showattr /filter:"(&(objectCategory=computer) PrimaryGroupID=516)" /subtree /atts:operatingSystem,operatingSystemVersion,operatingSystemServicePack dc1.example.internal “cn=dc1,ou=domain controllers,dc=example,dc=internal”


2. Run this on the Windows Server 2008 Server

To verify that all DC's have all replicas and are replicating consistently run the following command:

repadmin /replsum /bysrc /bydest /sort:delta


3. Run this on this on Windows Server 2003 Server with support tools installed

To determine whether the Sysvol share replica sets function correctly in each domain and various other test, run the following command from windows server 2003 support tools folder, which will create a subfolder with various log files.

Health_chk.cmd dc1 dc1.example.internal : Verify the FRS and replication on dc1


Health_chk.cmd dc1 dc2.example.internal : Verify the FRS and replication on dc2

4. Run this command on the Windows Server 2008 Server

DCDIAG.EXE /e /test:frssysvol /s:dc1 : To verify that dc1 domain controller has a shared Netlogon and Sysvol share

DCDIAG.EXE /e /test:frssysvol /s:dc2 : To verify that dc2 domain controller has a shared Netlogon and Sysvol share.


5. Run this command on the Windows Server 2008 Server

DCDIAG /test:FSMOCHECK /s:dc1 : To verify the FSMO roles and placement

DCDIAG /test:FSMOCHECK /s:dc2 : To verify the FSMO roles and placement

6. Run this command on the Windows Server 2008 Server

REPADMIN /SHOWREPS dc1 : Verify that the schema master and each infrastructure master have performed inbound replication of Active Directory since last booted.

REPADMIN /SHOWREPS dc2 : Verify that the other DC has per-formed inbound replication of Active Directory since last booted.


7. Run this command on the Windows Server 2008 Server

REPADMIN /SHOWUTDVEC dc1 dc=example,dc=internal : Verify the Up-To-Dateness Vector on domain controller DC1

REPADMIN /SHOWUTDVEC dc2 dc=example,dc=internal : Verify the Up-To-Dateness Vector on domain controller DC2.


8. Run this command on Windows Server 2008 Server DC and the Windows 2003 DC

The steps below is all about making sure that there are no problems in the current environment that could prevent us from doing the needed change. If a problem is found that could potentially be a showstopper during the change, it's of course better to address it before starting making changes to environment. A infrastructure problem in the environment is likely not going to vanish after the change, and can only block or slow down the change process. You don't either want to be halfway trhough the change process only to find out that you can't continue.

Check the following on the DC's.
• EVENTVWR.MSC and any erors in the System, Application, DNS, Directory Replication, FRS replication etc.?

• Physical connectivity, are all the network adapters correctly configured?

• Network connectivity, are we connected to the network?

• Name registration, are the DC name registered correctly in the DNS with correct IP-address. Any double registrations with names or IP-addresses?

• Name resolution, does the DNS name resolution work? Do we have correct information in the _msdcs zone and in each Site subdomain?

• Authentication, any authentication errors or warnings on the DC?

• Group Policy, any GPO processing problems in the eventlog?

• Security policy, any problems with the security subsystem?

• Disk subsystem, any problems with the disk subsystem?

• Schema, what schema version do we have on the AD and on Exchange?

• Topology, is the replication topology correctly configured?

• Replication engine, any issues with the replication, does manual replication triggering work?

9. Run this on the Windows 2003 DC

ntbackup dc2 before_Exchange2010prep : Create a SystemStateBackup which Protects the configuration and AD-database and enables a smooth restore operation. Select a local disk or to a network share as the target

10. Run this command on the Windows Server 2008 Server DC

Run a Windows Server Backup or similar backup product of the whole system parti-tion, or a least the system state. This example backups SystemState to a target disk F:
wbadmin start systemstatebackup -backuptarget:f:

11.Run this command on the Windows Server 2008 Server DC

repadmin /options dc1 +DISABLE_OUTBOUND_REPL : Disable Outbound replication on the schema master dc1 to dc2. This is supported, however disconnecting the network cable isn't a supported way or working way of preventing the changes made to dc1 from replicating to dc2

12.Run this command on the Windows Server 2008 Server DC

Before you start the installation of Exchange AD updates and preparation you need to make sure that the prerequisites for Exchange are installed on the Windows Server 2008 DC. This is a requirement since the command and setup used is leveraging the .NET framework and same setup tool as used by the full Exchange Server 2010 installation program. http://technet.microsoft.com/en-us/library/bb691354.aspx
Run from the command prompt and the media drive, example D:

d:\setup.com /preparelegacyexchangepermissions /DomainController:dc1 : Run on dc1 which prepares the Active Directory permissions for Exchange Server 2010, this operation will take a couple of minutes.

13.Run this command on the Windows Server 2008 Server DC

Check the output from the command for any errors or warnings : Check on dc1, if its not detailed enough, then Check the log file in the %systemroot%\ExchangeSetupLogs\ExchangeSetup.log


14. Run this command on the Windows Server 2008 Server DC

Run EVENTVWR.MSC : Check the event log for errors on dc1, System Log, Directory Service Log, Application Log, FRS Log, DNS log, Security Log

15.Run this command on the Windows Server 2008 Server DC

Run ADSIedit.msc : Verify that the previous command has:

Added an ACE to the domain root access control list (ACL) to provide the EES group with WRITE_PROP permissions on the Exchange-Information property set.

• Adds an ACE to the domain root ACL to provide authenticated users with READ_PROP permissions on the Exchange-Information property set.

• Adds an ACE to the AdminSDHolder container of the domain to provide the EES group with WRITE_PROP and READ_PROP permissions on the Exchange-Information property set.

• Adds an ACE to the Exchange organization container ACL to provide the EDS group with WRITE_PROP permissions on the Exchange-Information property set.

16. Run this command on the Windows Server 2008 Server DC

repadmin /options -DISABLE_OUTBOUND_REPL : Run this command to Enable Outbound replication again which will replicate out the changes made within one minute in site. You can also trigger a manual replication from Active Directory Sites and Services, or run a script that does triggers it.

17. Run this command on the Windows 2003 DC

Run ADSIedit.msc : Verify that it has replicated with the Windows Server 2008 DC and:

Added an ACE to the domain root access control list (ACL) to provide the EES group with WRITE_PROP permissions on the Exchange-Information property set.

• Adds an ACE to the domain root ACL to provide authenticated users with READ_PROP permissions on the Exchange-Information property set.

• Adds an ACE to the AdminSDHolder container of the domain to provide the EES group with WRITE_PROP and READ_PROP permissions on the Exchange-Information property set.

• Adds an ACE to the Exchange organization container ACL to provide the EDS group with WRITE_PROP permissions on the Exchange-Information property set.


18. Run this command on the Windows Server 2008 Server DC

repadmin /options dc1 +DISABLE_OUTBOUND_REPL : Disable Outbound replication on the schema master.


19. Run this command on the Windows Server 2008 Server DC

d:\setup.com /prepareschema /DomainController:dc1 : Run this on dc1 and this command will commit forest wide schema changes in the target Forest. This command can take 5-20 minutes depending on the needed updates and the hardware performance.

20. Run this command on the Windows Server 2008 Server DC

Check the output from the command : Check on dc1. If its not detailed enough, then Check the log file in the

%systemroot%\ExchangeSetupLogs\ExchangeSetup.log

 21.Run this command on the Windows Server 2008 Server DC

Run EVENTVWR.MSC : Check the event log for er-rors on dc1, System Log, Directory Service Log, Application Log, FRS Log, DNS log, Security Log.

22. Run this command on the Windows Server 2008 Server DC

Run Dsquery : Verify that it has updated the schema version: This command checks the Active Directory Schema version.

dsquery * cn=schema,cn=configuration,dc=example,dc=internal -scope base –attr objectVersion

Value  Schema Version
13        Windows 2000 Server
30        Windows Server 2003 RTM/SP1/SP2
31        Windows Server 2003 R2
44        Windows Server 2008 RTM
47        Windows Server 2008 R2


23. Run this command on the Windows Server 2008 Server DC

Run the Dsquery command below in one line : This verifies that it has updated the schema version: This command checks the Active Directory Exchange Schema. A value of 14622 indicates Exchange Server 2010 RTM schema, which is the same version Exchange 2007 SP2 uses.
dsquery * CN=ms-Exch-Schema-Version-Pt,cn=schema,cn=configuration,dc=example,dc=internal -scope base –attr rangeUp-per

Value   Schema Version
4397     Exchange Server 2000 RTM
4406     Exchange Server 2000 SP3
6870     Exchange Server 2003 RTM
6936     Exchange Server 2003 SP3
10628   Exchange Server 2007 RTM
11116   Exchange Server 2007 SP1
14622   Exchange Server 2007 SP2 and Exchange Server 2010 RTM

24. Run this command on the Windows Server 2008 Server DC

repadmin /options -DISABLE_OUTBOUND_REPL : Run this command to Enable Outbound replication again

ReplicateAll.cmd : Run this to triggers a full replication of the whole Forest and all domain controllers to speed up the operation and check for errors. Run this only if you have multiple sites to speed up replication. In a single site environment this is a redundant script to run since the replication is done within a minute.

25. Run this command on the Windows Server 2003 Server DC

Run the Dsquery command below in one line : This verifies that it has updated the schema version on the second Windows 2003 based DC: This command checks the Active Directory Exchange Schema. A value of 14622 indicates Exchange Server 2010 RTM, which is the same version and schema that Exchange 2007 SP2 uses.

dsquery * CN=ms-Exch-Schema-Version-Pt,cn=schema,cn=configuration,dc=example,dc=internal -scope base –attr rangeUp-per

26.Run this command on the Windows Server 2003 Server DC

repadmin /options dc1 +DISABLE_OUTBOUND_REPL : Disable Outbound replication on the schema master before next change.

27. Run this command on the Windows Server 2008 Server DC

d:\setup.com /prepareAD /DomainController:dc1 : Run this command on dc1 to commit forest-wide enterprise changes and also domain specific changes in the target Forest. Since it’s a Coexistence there’s no need to specify any organization name. This command usually takes a couple of minutes.

28. Run this command on the Windows Server 2008 Server DC

Check the output from the command for any errors : Check on dc1. If its not detailed enough, then Check the log file in the

%systemroot%\ExchangeSetupLogs\ExchangeSetup.log


29. Run this command on the Windows Server 2008 Server DC

Run EVENTVWR.MSC : Check the event log for er-rors on dc1, System Log, Directory Service Log, Application Log, FRS Log, DNS log, Security Log

30.Run this command on the Windows Server 2008 Server DC

Verify in Active Directory Users and Computers that it:

Created the following management role groups within the Microsoft Exchange Security Groups OU: Exchange Organization Management
Exchange Recipient Management
Exchange Server Management
Exchange View-Only Organization Management
Exchange Public Folder Management
Exchange UM Management
Exchange Hygiene Management
Exchange Records Management
Exchange Discovery Management
Exchange Delegated Setup

31. Run this command on the Windows Server 2008 Server DC

repadmin /options -DISABLE_OUTBOUND_REPL : Run this command to Enable Outbound replication again.

32. Run this command on the Windows Server 2003 Server DC

Verify in Active Directory Users and Computers that it has replicated:

The following management role groups within the Microsoft Exchange Security Groups OU:
Exchange Organization Management

Exchange Recipient Management
Exchange Server Management
Exchange View-Only Organization Management
Exchange Public Folder Management
Exchange UM Management
Exchange Hygiene Management
Exchange Records Management
Exchange Discovery Management
Exchange Delegated Setup

33. Run this command on the Windows Server 2003 Server DC

ntbackup dc2 after_prepareAD : SystemStateBackup Protects the configuration and AD-database and enables a restore operation if needed

34. Run this command on the Windows Server 2008 Server DC

Run a Windows Server Backup or similar backup product of the whole system partition, or a least the system state. This example backups SystemState to a target disk F:
wbadmin start systemstatebackup -backuptarget:f:


35. Multiple Domains (if needed)

If you have more than one domain you need to run the following command in each domain

d:\setup.com /prepareDomain : Run this command in each domain in the Forest to commit domain specific changes to support Exchange Server 2010.

This concludes the AD preparation steps. You are now prepared for the next step, the Exchange Server 2010 installation.

Microsoft Learning: Exchange Server 2010 Courseware

       
New Courses
Microsoft recently released a new Exchange Server 2010 course, 10135A

http://www.microsoft.com/learning/en/us/course.aspx?ID=10135A&Locale=en-us

This is a very comprehensive course covering most topics of the Exchange Server 2010 product to a decent administration depth; I would say 200-300 level. It’s obviously not near as deep as the 3-week Exchange Server 2010 Master Class, but then again that’s not the goal or target audience either.

A student taking this course can probably do most of the day to day Exchange Server 2010 administration, but it will not be enough as a preparation for doing a complete migration/transition from previous Exchange version, without digging deeper into the migration stuff, or hire a good consultant for the job.

There’s also another great course in the Microsoft Learning pipeline, 10233A, Designing and Deploying a messaging solution. It’s still in Beta version though, due to release later this year. I would say it’s a very good resource for consultants or administrators that are designing a Exchange Server 2010 deployment, migration, coexistence etc. It’s utilizing the MSF/MOF framework which of course is a practical implementation of the ITIL foundation, processes and practice. Remember, it’s all about technology, people and processes.

I've seen that higly educated people using the latest and greatest technology still fail due to lack of processes and procedures.

Great Quality
I’ve been working as an MCT for the last 14 years and seen a lot courseware during those years. Generally it’s been very good courseware, with an exception the last couple of years with the introduction of the new format in Windows Vista/2008 courseware. Microsoft Learning quality processes have improved the courseware since then, and there are now back on a very good quality level I would say.

The Exchange courseware has always been good and even the courseware created during Vista/Windows Server 2008 timeframe is decent. I’m talking about the Exchange Server 2007 courseware 3938, 5047, 5049, 5050 and 5051.

The new course 10135 is at the same level as the Exchange Server 2003 course 2400, or the Exchange Server 2000 course 1572 and 1573 which are great. There is still a bit room left for quality improvements to the almost flawless Exchange Server 5.5 courses, 1026 and 973.

Good Resource
I think the 10135 and 10233 courses are great resources for any consultant/administrator working with Exchange Server 2010. If you have the chance, do take the class or just pick up and read the course material. I have read them and being an Exchange Master I still enjoyed and learned something from them, especially the 10233 course.

MCM: Microsoft Certified Master on Microsoft Exchange Server 2010

Unique Certification

Me and 12 other guys from around the world recently spent 3 weeks (from the 1st to 20th of March 2010) in Redmond for the 5th master rotation on Exchange Server, the first on Exchange Server2010. Out of us 13, five managed to pass the three written exams and the practical qualification lab at first attempt and became certified masters. http://blogs.technet.com/themasterblog/archive/2010/03/29/the-first-5-mcm-exchange-2010-s-ever-in-the-world-just-them-only-them.aspx

Great Training

The training was just awesome, the best I had during my 25 years as an IT professional. We had the very best instructors, deep dive course material and real life practical labs with our own complete lab environment. The Microsoft training facilities are also top notch with nice classrooms. The investment is definitely worth it, you can’t get this kind of training anywhere else and the return you get in knowledge and the community resource is priceless.

Dedication and a lot of Hard work

I knew that the training was going to be tough from reading the master blog, it was even tougher than I expected. We were in the classroom at 7.30 every morning and the lectures started at 8. We continued the whole day until 6pm on average, some days even to 8pm. After the lectures and labs, we usually stayed a couple of more hours for more labs and testing. Saturdays were lab days, with 12h of optional labs, but much needed for the upcoming qualification lab. Sundays were study days for the written exams on Monday mornings. I spent around 300h during this training listening, reading, doing labs and taking exams. There was no time to do anything else, but focusing on Exchange Server 2010 for the 3 weeks. I didn’t even turn on the TV once during the 3 weeks of training, there was only time to make some short phone calls back home. I hardly answered any emails during this period either. I think this focused effort and dedication made me successful in passing the entire program at first attempt.

The Community

Becoming an Exchange Master opens up the door the exclusive community of the Exchange Product Team and all the certified Exchange Rangers/Masters. This is probably the most valuable resource for an Exchange consultant and IT pro in the world. Any Exchange question you might have is most likely going to be answered in depth and with authority of this community. All Masters are under NDA which also means that we get to know what’s coming in the future releases, but we can’t talk or blog about it until the information is officially released.

Sharing knowledge

The intention of this blog is to share my knowledge and experience from consultancy and training work in the field. The communities and the web has helped me a lot during the years, and I too feel like contributing to the world’s greatest resource.