Reconnecting Disconnected Mailboxes and Workstations (Windows 2003)

Recently I wrote about Transparent Windows Workstation Domain Migration. Through some unfortunate circumstances (can you say “array controller failure”) I came to realize, that the process to re-join a workstation to a re-built domain is almost identical.

In case of this situation I had fortunately backed up all the data very well, and virtually nothing was lost (except for a ~week’s worth of time in rebuilding the system) as the result a hardware failure. I had also backed up the Windows 2003 Server (Standard edition) system state, but elected not to use it because the hardware had changed sufficiently (I chose not to use the same kind of array controller in the restored system) so that restoring the registry as a part of the system state restore would have likely messed things up. As a result the newly created fifteen or so user accounts were no longer in touch neither with the restored Exchange 2003 mailboxes (even though the user names matched), nor with the workstations that were seemingly on the correct domain because the unique user IDs, or SIDs, had changed.

To re-join the workstations to the re-built domain forest (i.e. one that was not restored from the system state backup), you need to un-join them first, either by joining a workgroup (name doesn’t matter, you can call it just a generic “WORKGROUP”, the point is to leave the domain), or by using a free Unjoin utility. Then restart, and proceed with my Transparent Windows Workstation Domain Migration instructions. The only thing of note is that the user profile name after re-joining the domain is likely going to be of format username.DOMAIN.000, unless the original profile folder name lacked the domain suffix, in which case you’ll find the newly created profile at username.DOMAIN. Either way it may be a good idea to restore the original profile folder name as described in steps 15-16 of the Transparent Windows Workstation Domain Migration instructions.

What about the disconnected Exchange 2003 accounts, you say? Before I started the restore I blocked SMTP port (25) at the firewall to prevent mails from being received during the restore (DynDNS Mailhop Backup MX service is utilized to buffer arrived emails while the server is unavailable), then backed up the mail that had accumulated in last two days (since the server was restored to service) using ExMerge to be appended to the restored mailbox store. Obviously, restore overwrites anything arrived since the restore took place.

I found lots of good information on the web about how to accomplish this (most notably, once again, an article at Petri IT Knowledgebase: How can I recover a deleted mailbox in Exchange 2000/2003) once I had realized that the Administrator has to have Send As / Receive As privileges for the mailbox store(s) being restored (and if you’re not using the system admin account, the appropriate privileges must be delegated to that account), and that the mailbox store(s) must be unmounted before NTBACKUP restore is started (NTBACKUP wants to mount them, so if the restore fails to start for some of the many potential reasons and you retry, you’ll need to unmount the mailbox stores again first), and that “This Database can be overwritten by a restore” must be checked in mailbox store > Properties > Database, and that “Last Restore Set” must be selected in NTBACKUP (or else you’ll get an error “.. it is a database restored from a backup set on which hard recovery was not started or did not complete successfully”, and you have to issue a command line command eseutil /cc [path to directory containing Restore.env that was created during the restore to a log folder you selected] after restore has been completed), assuming, of course, that you’re not restoring incremental sets in which case you want to check the “Last Restore Set” when restoring, well, the last restore set. ๐Ÿ™‚ If NTBACKUP restore fails, don’t expect any kind of information, other than “failed” and perhaps some rather cryptic error message under “Report”. Application events (in Event Viewer) generally yield better clues as to what’s not right. Also note that the process is somewhat different for Exchange 2007 as described in this MS TechNet article.

Overall, I must say the mailbox restore process wasn’t really that straightforward and certainly not enjoyable, and I came to wonder on multiple occasions through the process whether this really was the easiest way Microsoft could have implemented it, or whether the process was intentionally designed to be so convoluted that generally only the people with MCSE know it by heart (after having prepped for the exam, that is ๐Ÿ˜‰ ).

“External” cause for Outlook 2003 startup crash

Today I debugged Outlook 2003 startup crash on a PC with Windows XP SP2, and no known changes/updates applied overnight (Outlook was working in the evening, and started crashing in the morning with no apparent cause). I started with the obvious รขโ‚ฌโ€ I created a new profile in Settings > Control Panel > Mail, and pointed it to an external POP3 account to exclude the possibility that the Exchange server would somehow induce the crash. No dice, crashed again. Somewhere on the web I found the suggestion to close Outlook, then delete outcmd.dat found in Documents and SettingsusernameApplication DataMicrosoftOutlook, and restart Outlook. No luck, crash. So it must be a corrupted installation, right? I uninstalled Office 2003 (and deleted any remaining files in c:Program FilesMicrosoft OfficeOFFICE11, reinstalled, applied SP2 and any subsequent hot fixes, and created a fresh profile. CRASH!! At this point I was running out of ideas, and were already contemplating OS reinstallation as the last resort. Fortunately the good people at Symmetrix Technologies had the saving suggestion: the system has Cloudmark anti-spam software installed, and Cloudmark had recently posted a big red alert (see below) on their support page. So I deleted the SpamNet folder as instructed on the Cloudmark support page, and the troubled Outlook installation was back in business.

Outlook crash alert on the Cloudmark site

Transparent Windows Workstation Domain Migration

Couple of days ago I was faced with a task to move a small office environment into a new Windows domain. Basically an old LAN server was replaced with a new one, and at the same time the domain was renamed and restructured in a more logical way.

Switching the workstations (all running Windows XP Pro) to the new domain proved to be a bit of a challenge, especially since I wanted to preserve the users’ existing profiles. Lots of web searching ensued, and eventually I found a few resources such as this, this, and this. Especially the first of the three sites (Recovering a Windows Profile by Drew McLellan + reader comments, especially that by Carl Farrington) gave a very good starting point. Below is a cohesive step-by-step list, composed from the aforementioned resources with some new ideas and clarifications added.

The instructions below assume that you have a new Domain Controller ready, and also that you are somewhat familiar with the associated menus (i.e. not everything is spelled out). These instructions were written for Windows XP (Pro), but should work with other Windows flavors with minor modifications.

  1. Before you begin, save (or make a note of) each workstation’s desktop background image (if you care). For some reason it was not automatically reinstated after the new domain was joined (in fact, as far as I could tell it was the only bit of a configuration that was lost in the process).
  2. Make sure you have the local administrator password for all of the workstations (any local account with admin privileges that is not going to be joined to the domain is ok). If you don’t have admin password but the account that you have a password for has admin privileges, you can go to Settings > Control Panel > User Accounts and reset the password. If you’re at loss รขโ‚ฌโ€œ no access to an account with admin privileges รขโ‚ฌโ€œ refer to Petri IT knowledgebase article about the subject.
  3. Make sure the user profile you’re going to migrate is a LOCAL PROFILE (in Settings > Control Panel > User Accounts). If it is not, make it local (roaming profiles stand to lose data during the migration process and these instructions do not apply to them!).
  4. Write down the name of the user’s profile folder; it’s usually “c:Documents and Settingsusername” or “c:Documents and Settingsusername.OLDDOMAIN”

  5. Join the new domain (you know, Settings > Control Panel > System > Computer Name > Change > Member of.. Domain [enter domain name] > enter domain administrator username/password if requested > reboot).
  6. Log in as the user (with the password set in the Domain Controller). This creates the new profile folder at “c:Documents and Settingsusername.NEWDOMAIN

  7. Restart, then login as the local admin (or as any local user with administrative privileges).
  8. Move the newly created user profile folder someplace else (doesn’t matter where… the folder can also be deleted, but I’d keep it until the process is complete, just in case ๐Ÿ™‚ ). In other words, move the new profile folder at “c:Documents and Settingsusername.NEWDOMAIN”, for example, to “c:tempusername.NEWDOMAIN“, or just delete it (if you’re brave).
  9. Rename the old profile folder to the new name, i.e. usually from “c:Documents and Settingsusername” or “c:Documents and Settingsusername.OLDDOMAIN” to “c:Documents and Settingsusername.NEWDOMAIN”. In an event that the original user profile folder is locked and can’t be renamed, use the free Unlocker tool to identify the locking process, and to unlock it (you can run unlocker on the entire profile folder and it’ll recurse through all files).
  10. Add the new domain user to the profile folder’s filesystem permissions, and give full control for that user. Highlight the folder that you just renamed, right click on it > select Properties > Security. You should see the old profile (now displayed as string of number and letters), marked with a question mark. Click on it to select it, and select Remove to remove the old profile. Then click Add, and enter the new domain username (as NEWDOMAINusername) as an authorized user for the folder and for all the content in it. Click OK. With the newly added user highlighted check Full Control “Allow” checkbox, then click Advanced > check “Replace permission entries on all child objects..” > click OK, then “Yes”. Depending on amount of content it could take a few moments for the system to parse through all the files and folders in the profile folder as the permissions are being updated.
  11. Add the new domain user as a local user (and give the user administrative privileges for their computer if desired). Go to Settings > Control Panel > User Accounts > Add > Browse > enter the domain username as NEWDOMAINusername, and click Check Names. Depending on the setup (and the account you’re logged in as), you may be prompted for the domain administrator’s user/pass. The name is validated (it becomes underlined if it’s ok), then click on OK, then Next. Unless your policy somehow restricts it, you should add the domain admin as a local admin user at this point with this same procedure.
  12. Adjust registry. If you added the domain admin as an local administrative user, you may log off and log back in as the domain admin. Otherwise proceed as the local administrator (the difference is you may need to enter domain admin username/password when you add or validate user names).

    a. Make sure all files are visible (i.e. open My Computer > Tools > Folder Options > View > select “Show hidden files and folders”, and uncheck “Hide extensions for known file types” and uncheck “Hide protected operating system files”).
    b. As a precaution make a backup copy of the user’s NTUSER.DAT file. Go to the user’s profile folder (c:Documents and Settingsusername.NEWPROFILE) and copy the NTUSER.DAT to NTUSER.DAT.BAK. You can, of course, skip this step, but better be safe than sorry :).

    c. Run registry editor (Start > Run > regedit)

    d. Highlight HKEY_USERS, then select File > Load hive

    e. select NTUSER.DAT that resides in the user’s profile folder (i.e. “c:Documents and Settingsusername.NEWDOMAINNTUSER.DAT”), click Open, and give it any easily identifiable name (doesn’t matter what รขโ‚ฌโ€œ this is just a temporary name).

    f. expand HKEY_USERS and highlight the subkey (the name you gave it in the previous step). Right click, and select Permissions. There should be a SID (user ID) with a question mark indicating “unknown”. Delete it. Then click Add, and enter the domain user’s name as in NEWDOMAINusername. Click on Check Names (if you’re logged in as the domain admin the name will be validated immediately, otherwise you’ll be asked for the domain admin username/password first). If the name is ok, it’ll be underlined, then click OK.

    g. highlight the domain user you just added, then check Full Control under “allow”, then click Advanced, check “Replace permission entries on all child objects…” and click OK. If “not all child objects could be updated” (or such) error is displayed, ignore it. Click OK.

    h. with the same subkey (that you created few steps earlier) still highlighted, select File > Unload Hive, then select “Yes”, and close the registry editor.

  13. Log off, then log in as the domain user.

  14. Go to “c:Documents and Settings” and highlight the user’s own profile folder (again, usually “c:Documents and Settingsusername.NEWDOMAIN“). Right click > Properties > Security > Advanced > Owner. Select the user from the list and check “Replace owner on subcontainers and objects” and click on OK. Again, this could take a few moments to complete. Then click on OK to close the user profile properties.
  15. Restart, then log in as a user with local administrative privileges (but not as the user we’re migrating, even if you assigned admin privileges to them). Go to “c:Documents and Settings” and rename the user’s profile folder back to what it was originally called (i.e. the name you wrote down in step 4). This step might result in a confusing profile folder name if the old domain was part of the profile folder’s name (i.e. username.OLDDOMAIN), however, if the user has installed lots of 3rd party software that have saved data in the user’s profile (such as in user’s My Documents, or Application Data folders), all of those programs’ setting will need to be adjusted to reflect the changed profile folder name unless the profile folder is renamed to what it was before the migration.
  16. Start Registry Editor (Start > Run > Regedit), and navigate to HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionProfileList. Once there, go through the profile subkeys (S-1-5-, etc.) and look for the profile whose “ProfileImagePath” is set to that of the user. It should have a value such as “%SystemDrive%Documents and Settingsusername.NEWDOMAIN“. Double-click on the ProfileImagePath and set the value to reflect the original profile folder name as renamed in step 15 (e.g. “%SystemDrive%Documents and Settingsusername“, or “%SystemDrive%Documents and Settingsusername.OLDDOMAIN“). Then close the Registry Editor.

You’re done! You might want to finally log in as the user to make sure everything is working as they should. Finally, since the user’s old password was set by the new domain, and since you’ve logged in as that user few times, you might want to reset the user’s password in the domain controller with “User must change password at next logon” requirement.

Disclaimer: While this worked for me, various system settings may affect the described steps. If you destroy someone’s profile with 700Gb of irreplaceable data in the process, it’s not my fault ๐Ÿ™‚ , so be careful, and preferably take a full backup before you begin. While this process allows profile migration without having to duplicate it, it’s a good idea to make a backup copy of the profile (if the disk space allows it) before you migrate it; you can then delete it once the migration has been completed successfully.

I welcome any comments, corrections, or other feedback.