This issue was a real booger and I almost threw in the flag and called in the big guns.
This error has been around for awhile. There is a lot of information out there on it and a LOT of reasons it can occur.
I've actually run across this now twice. Once in my server VM environment and also on new desktops. It is possible / likely they are related due to the use of switches being simular / same.
In this post I'm focusing on the Virtual Environment issue.
I first discovered the issue when building an Exchange 2010 server and finding that the services where not starting automatically on boot. This led me to find the Netlogon 5719. After a review of the events it was obvious that this service was attempting and failing to start before the network was connected.
After find this: http://support.microsoft.com/kb/938449 I tried some of the suggestions with no help. Note this setup was with ESXi 5.1 going back to HP ProCurve switches (2810's). STP was off on the switches. Also, connected to the same switches is a XenServer environment and a few physical servers which do not see the issue.
Some of the different posts and KB's I found suggested that this isn't an issue and can safely be ignored as long as you can reach the DC to login. After the set timeperiod Group Policy will apply. Unfortunately this is NOT a solution nor a good workaround (for desktops, servers, anything). This causes lots of issues in a domain environment especially where folder redirection, logon scripts, etc. The proper fix is to be able to get the NIC to initialize before netlogon OR for MS to provide a method for admins to reliably force netlogon to wait for the NIC.
After messing around for awhile I discovered that this only occurs if the NIC is set to static IP. When set to DHCP all works as expected.
So, at this point we could do DHCP reservations to make it work, BUT this isn't a solution for DC's or DHCP servers, and sometimes a static address is necessary or easier.
After finding a thread on VMWare communities that was exactly my issue it was suggested to try changing the ArpRetryCount.
http://communities.vmware.com/thread/316237?start=15&tstart=0
Bingo!
This could indicate a deeper network issue or possibly a flaw in logic as to when netlogon service should attempt to start.
Note: I also commonly see an issue very simular to this on workstations with SSD's (some differences, occurs when set to DHCP but not static, etc). In these cases changing the ArpRetryCount does not help although I did find that it is heavily dependent on the type of switch that the workstation is plugged into. For instance, the issue occurs when plugged into HP ProCurve switches, but does not occur when plugged into cheapo Linksys / Cisco switches. This likely indicates configuration issue with HP ProCurve (although, many report same or simular issues with enterprise Cisco switches). It may also be caused by the type of NIC / driver on the system (ie Realtek driver issue). I have not been able to dig into this issue in great detail yet.
Tuesday, April 16, 2013
Citrix print management service crashing
Awhile back I posted about cleaning up print drivers in XA4.5. Recently I started to have issues again with printers not being auto created with what appeared to be the same issue of the spooler crashing and taking the Citrix service with it. Oddly it wasn't logging the crash though.
(see this post: http://didyourestart.blogspot.com/2009/04/terminal-server-citrix-printing-errors.html)
It then occurred to me that it's not the same issue! Duh
On a pool of 8 XenApp 5.0 (windows 2003) with R07 installed I've found that occasionally the Citrix Print Management Service will crash. Note that in this instance the Print Spooler is NOT crashing, only the Citrix Service. I determined this by using the script in my other printing issue post so that it would log when the print spooler crashed. In this case no log was ever generated on the servers after a failure.
I then added a new short script and set it to run on failure of the Citrix Print Management Service. On the next failure sure enough I had my log showing the failure time.
Batch File contents
net start "Citrix Print Manager Service"
SET logfile=C:\AdminTools\CitrixCrashLogs.log
ECHO Citrix print management service crashed on %date% at %time% on %computername% >> %Logfile%
I then set the Citrix Print Management Service to run this program on failure.
This tells me that it's a different issue causing the failure of the Citrix service since print spooler isn't actually crashing. I believe this is an issue that was introduced sometime post R05 as I never had the issue (that I'm aware of) until updating to R07. Note that I had skipped installing R06. I also tested this on a fresh Citrix build with the same results.
I now implement the above batch file as part of my build on all XenApp servers. This has reduced the help desk calls for this issue down to 1 or less a quarter.
(see this post: http://didyourestart.blogspot.com/2009/04/terminal-server-citrix-printing-errors.html)
It then occurred to me that it's not the same issue! Duh
On a pool of 8 XenApp 5.0 (windows 2003) with R07 installed I've found that occasionally the Citrix Print Management Service will crash. Note that in this instance the Print Spooler is NOT crashing, only the Citrix Service. I determined this by using the script in my other printing issue post so that it would log when the print spooler crashed. In this case no log was ever generated on the servers after a failure.
I then added a new short script and set it to run on failure of the Citrix Print Management Service. On the next failure sure enough I had my log showing the failure time.
Batch File contents
net start "Citrix Print Manager Service"
SET logfile=C:\AdminTools\CitrixCrashLogs.log
ECHO Citrix print management service crashed on %date% at %time% on %computername% >> %Logfile%
I then set the Citrix Print Management Service to run this program on failure.
This tells me that it's a different issue causing the failure of the Citrix service since print spooler isn't actually crashing. I believe this is an issue that was introduced sometime post R05 as I never had the issue (that I'm aware of) until updating to R07. Note that I had skipped installing R06. I also tested this on a fresh Citrix build with the same results.
I now implement the above batch file as part of my build on all XenApp servers. This has reduced the help desk calls for this issue down to 1 or less a quarter.
Friday, March 1, 2013
Convert XenServer XVA to VMDK for VMWare ESXi 5.1
During our conversion from XenServer to VMWare I had one machine that had a woopsy and would no longer boot in XenServer without BSOD. Of course, this is the ONE machine I didn't snapshot first HA.
Environment:
XenServer 5.6 SP2
ESXi 5.1
Cause:
During the conversion, I uninstalled the XenServer tools. When I went to use VMConverter on the machine it couldn't find the disk due to drivers on the SCSI controller. So, I went to select a generic driver for the controller and low and behold, I selected the wrong one and rebooted. ACK Of course I was greeted by the BSOD.
Solution:
I could have just rebuilt the machine, but it was one of those pain machines (ie, reconfiguring it would be more painful then spending an hour seeing if I could fix it).
Your results may vary :)
Environment:
XenServer 5.6 SP2
ESXi 5.1
Cause:
During the conversion, I uninstalled the XenServer tools. When I went to use VMConverter on the machine it couldn't find the disk due to drivers on the SCSI controller. So, I went to select a generic driver for the controller and low and behold, I selected the wrong one and rebooted. ACK Of course I was greeted by the BSOD.
Solution:
I could have just rebuilt the machine, but it was one of those pain machines (ie, reconfiguring it would be more painful then spending an hour seeing if I could fix it).
- Export from XenServer to XVA format
- I tried to export to OVF, but it would fail and after some quick looking on the citrix forums it looked like it would likely be easier to export to xva then convert to ovf
- XenConvert v2.3.1 (version 2.5 doesn't have the options necessary to do this)
- From = Xen Virtual Appliance
- To OVF
- This converts to OVF format which gives you 2 files, an OVF and a VHD
- WinImage v8.5 (http://www.winimage.com/download.htm)
- Disk dropdown
- Convert virtual hard disk image
- Select the OVF (vhd file)
- OK
- Type name and change save as type to vmdk
- When you click save the conversion starts
- At the end you don't need to mount it with winimage
- Create a new virtual machine using the datastore that you want.
- Edit the VM and delete the Hard Disk
- Browse the datastore and delete the vmdk file
- Veeam FastSCP (I used the older version)
- Copy the 2 files that WinImage created to the datastore VM Folder
- both are vmdk files. One will be large the other small. Both are required for this to work (otherwise when you add a hard disk in the next step it won't see it)
- Back in VMWare now
- Add hard disk
- Browse to the VMDK and select it
- Boot
- Login and install tools
- Restart
- Change the disk from IDE to SCSI
- In VMWare edit the VM and add a new hard disk of 1GB using SCSI (this pulls the controller into the image)
- Delete the disk you just created
- shutdown the VM
- Now we have to edit the vmdk, this can be done using vi, but I'm a windows guy so I used notepad++ on my workstation
- pull a copy of the small vmdk down to your local drive using Veeam FastSCP
- edit with notepad++
- modify the line "ddb.adapterType = "ide" change to "lsilogic"
- save and push the file back up to the datastore overwriting
- Delete the primary IDE hard disk
- Browse the datastore and delete the 1GB vmdk that we created earlier (for getting the LSI controller installed)
- Add new hard disk and point to the vmdk
- It will find it as SCSI / LSI Logic
- Note: if you have a CDROM, it will be IDE 0:1, so you'll want to delete it and re-add it so that it picks up IDE 0:0
- Boot
Your results may vary :)
Thursday, February 28, 2013
Convert from XenServer 5.6SP2 to VMWare ESXi 5.1
Recently we converted our main environment from XenServer 5.6 SP2 (Lefthand Networks SAN/iQ) to VMWare ESXi 5.1 (Nimble Storage CS240).
First off, we looked very hard at XenServer 6, Hyper 2012, and ESX. After getting hands on for each solution VMWare imo was hands down easier to use and gave better results.
Next, Nimble Storage is awesome! If your looking at storage give them a good look.
The conversion is pretty easy really, but if you get the steps wrong you can end up with BSOD and other ickyness. This worked for me, your experience may be different. I took snapshots at the Lefthand level and XenServer level before touching anything. Results may vary.
There may be an easier way, doesn't really matter to me. This worked consistantly for me so I'm stickin to it.
It's best practice to rebuild rather than convert. I only converted machines that couldn't be rebuilt, where being replaced soon (but not ready to replace just yet), or when I was short on time and had to move it immediately.
Server 2008 / 2008 R2
Since typing the Windows 2008 section, I tried something new that worked amazingly well with little downtime. I did this with Windows 2008 RTM x32 and Windows 2008 R2 successfully.
Windows 2008 RTM: I also had to delete the NIC (which was listed as Flexible) and add a new one for VMXNET3.
One final strange thing I noticed is that the IntialKeyboardIndicators key would get messed up.
This is found under KHEY_USERS\.Default\Control Panel\Keyboard
It would be set to 21474836648 after conversion
Changing this back to 0 made it work as expected.
First off, we looked very hard at XenServer 6, Hyper 2012, and ESX. After getting hands on for each solution VMWare imo was hands down easier to use and gave better results.
Next, Nimble Storage is awesome! If your looking at storage give them a good look.
The conversion is pretty easy really, but if you get the steps wrong you can end up with BSOD and other ickyness. This worked for me, your experience may be different. I took snapshots at the Lefthand level and XenServer level before touching anything. Results may vary.
There may be an easier way, doesn't really matter to me. This worked consistantly for me so I'm stickin to it.
It's best practice to rebuild rather than convert. I only converted machines that couldn't be rebuilt, where being replaced soon (but not ready to replace just yet), or when I was short on time and had to move it immediately.
Server 2008 / 2008 R2
- Download and install VMWare Converter 4.3, yes, the older version
- Disable any services necessary (ie, IIS, etc)
- Ensure your logged in through the default view, not RDP.
- Uninstall XenTools and reboot
- Go into Device Manager
- You'll see that the SCSI Controller doesn't have a driver.
- VMWare converter won't see the disks because of this
- Right click the SCSI Controller
- Update Driver Software
- Browse my computer for driver software
- Let me pick from a list of device drivers on my computer
- (Standard IDE ATA/ATAPI contoller)
- IDE Channel
- If you get the wrong one you'll likely see a BSOD upon reboot
- Reboot
- Open VM Converter
- Convert Machine
- Select "This local Machine"
- Note that "View source details..." lights up. Click it
- Ensure that a Source disk is listed (if you didn't change the controller driver then none will be listed and it will error when you attempt to convert)
- Type in the info for one of your VMWare hosts
- Select your datastore target
- Change RAM, CPU, etc as fit
- Finish and wait
- Once it's completed shutdown the VM in XenServer
- In the VMWare console edit the VM.
- Delete the CDROM and Hard Disk
- Add a new Hard Disk as the SCSI 0:0 and point to the VMDK
- Add new CDROM with basic settings
- Start the machine and install tools
- Note that the VM Version is listed as 4
- Shutdown the VM
- Right click the VM and choose the option for "Upgrade Virtual Hardware"
- It should now show as a vmx-09
- Change the nic to vmxnet3 if desired
- Boot and change IP address if needed
- Uninstall VMWare converter
Since typing the Windows 2008 section, I tried something new that worked amazingly well with little downtime. I did this with Windows 2008 RTM x32 and Windows 2008 R2 successfully.
- Download and install VMWare Converter 4.3. New version may work better.
- Open VM Converter
- Convert Machine
- Select "This local Machine"
- Type in the info for one of your VMWare hosts
- Select your datastore target
- I had to edit the devices and change the controller to IDE
- Finish and wait
- At this point it's extermely important to remember that we don't want both VM's on at the same time. BUT I wanted to ensure that my new VMWare VM would boot...
- Change Settings
- Change network to an isolated network off production.
- Delete the CDROM and Hard Disk
- Add a new Hard Disk as the SCSI 0:0 and point to VMDK
- Add new CDROM with basic settings
- Start the machine
- Uninstall XenServer Tools
- Reboot
- Install VMWare Tools
- Shutdown
- Note that the VM Version is listed as 4
- Shutdown the VM
- Right click the VM and choose the option for "Upgrade Virtual Hardware"
- It should now show as a vmx-09
- Boot the server and ensure it boots
- Shutdown VMWare VM
- Shutdown XenServer VM
- Edit VMWare VM and change NIC to production network
- Boot and change IP address if needed
- Uninstall VMWare converter
Windows 2008 RTM: I also had to delete the NIC (which was listed as Flexible) and add a new one for VMXNET3.
One final strange thing I noticed is that the IntialKeyboardIndicators key would get messed up.
This is found under KHEY_USERS\.Default\Control Panel\Keyboard
It would be set to 21474836648 after conversion
Changing this back to 0 made it work as expected.
Pagefile on XenApp
Building out a new environment and I got to the point of setting the pagefile...
I've always been told 1.5 x RAM or other numbers that I didn't know any better and just did it.
So, building the XA65 server and I look around to see what recommendations are for XenApp 6.5 and found this:
http://blogs.citrix.com/2011/12/23/the-pagefile-done-right/
What a concept, pagefile should be set depending on what the server load is. All these years and I only just now learned this.
I've always been told 1.5 x RAM or other numbers that I didn't know any better and just did it.
So, building the XA65 server and I look around to see what recommendations are for XenApp 6.5 and found this:
http://blogs.citrix.com/2011/12/23/the-pagefile-done-right/
What a concept, pagefile should be set depending on what the server load is. All these years and I only just now learned this.
Thursday, February 21, 2013
DHCP Migrate from Split Scope to Windows 2012 DHCP with Failover
There are already several excellent posts out there about how to migrate to Windows 2012 DHCP.
Microsofts own blog is an excellent reference.
http://blogs.technet.com/b/teamdhcp/archive/2012/09/11/migrating-existing-dhcp-server-deployment-to-windows-server-2012-dhcp-failover.aspx
I'm writing this, because I didn't find a lot about migrating from multiple DHCP servers with split scopes to a single 2012 with failover and bringing the leases along with. This is a very easy process and requires minimal work. (note, you can also just set conflict detection and then unauthorize the old servers and let it all sort itself out). I decided not to use conflict detection this time around as last time I did this some users got a popup about their IP expiring, this caused a call to IT to tell them to restart. Not a big deal, but this is easy and causes one less call to IT - I'm all for less calls.
In my case I migrated off 2 Windows 2008 R2 servers with split scopes. I wanted to pull the leases off both servers and combine them and then make the DHCP servers failover with hot standby.
I'll refer to the old servers as DC01 and DC02
The new servers will be DC03 and DC04
Now lets setup failover / replication
That's it. Now if you right click on a scope and go to properties you will find a failover tab. This will tell you if it's in failover and what the role of that particular server is.
At this point I would run netscan again and check which servers it sees handing out DHCP as well as which servers it shows as AD-authorized. In my case it shows DC03 as handing out DHCP and it shows AD authorized as DC03 and DC04.
http://www.softperfect.com/products/networkscanner/
Note: dhcploc.exe is included with the Microsoft tools on the Windows 2003 cd. (believe it's also on the 2008+ cd's). To run it I typed "dhcploc.exe mymachinesIPaddress". I took awhile to start displaying.
Microsofts own blog is an excellent reference.
http://blogs.technet.com/b/teamdhcp/archive/2012/09/11/migrating-existing-dhcp-server-deployment-to-windows-server-2012-dhcp-failover.aspx
I'm writing this, because I didn't find a lot about migrating from multiple DHCP servers with split scopes to a single 2012 with failover and bringing the leases along with. This is a very easy process and requires minimal work. (note, you can also just set conflict detection and then unauthorize the old servers and let it all sort itself out). I decided not to use conflict detection this time around as last time I did this some users got a popup about their IP expiring, this caused a call to IT to tell them to restart. Not a big deal, but this is easy and causes one less call to IT - I'm all for less calls.
In my case I migrated off 2 Windows 2008 R2 servers with split scopes. I wanted to pull the leases off both servers and combine them and then make the DHCP servers failover with hot standby.
I'll refer to the old servers as DC01 and DC02
The new servers will be DC03 and DC04
- Add the DHCP role to both of your new servers.
- On DC03 right click, run as administrator on Powershell.
- Export-DhcpServer -ComputerName DC01 -Leases -File C:\export\dhcpDC01.xml -Verbose
- Export-DhcpServer -ComputerName DC02 -Leases -File C:\export\dhcpDC02.xml -Verbose
- Open both xml files with notepad (I prefer Notepad++)
- These xml files have the full DHCP config for your old server, we need to combine the leases so that when you import all of the leases get combined.
- Do a search for
- This is the begining section for all your leases.
- On the DC02 xml file copy from the first (note that I dropped the "s") down to the last Lease section. Make sure you get both the starting and ending tags for each active lease.
Remove any leases that are from reservations if you have any, otherwise they will be duplicates. - In the DC01 xml file paste these leases into the section. I pasted mine after the last DC01 lease, but just before the tag.
- On DC03 run: Import-DhcpServer -ComputerName DC03 -Leases -File C:\export\dhcpDC01.xml -Verbose
- Ensure that DC03 is authorized
- In the DHCP mmc "unauthorize" the server for DC01 and DC02
- You may get a parameter error when unauthorizing servers. I found that it still worked if I closed the mmc and reopened. In one case I did the unauthorize from the server directly or simply waited a couple minutes then closed / reopened and did it again. Don't forget to use refresh.
- In a pinch you can also shutdown the DHCP Server service.
- Close DHCP mmc and reopen. Right click DHCP and Manage Authorized Servers and check that the list is correct.
- Go into your scopes and "merge" your scope exclusions. I had my new single scope exclusions already written down so that I could just delete all the old exclusions and re-enter the new ones. EDIT: You can also modify the xml file the same way you did for the leases to include the new exclusion ranges. Look for the tags
- I always like having conflict detection, so ensure this is on. It's under IPv4, Properties, Advanced. (if you had it set on your old DHCP then it will have migrated with the import)
- I like to use a tool like MS Windows dhcploc.exe or netscan to check and ensure that I don't have rogue DHCP servers. Check and ensure that only your new server is handing out addresses at this point. http://www.softperfect.com/products/networkscanner/
- 1 server 2012 handing out DHCP for the entire environment (DC03)
- 1 server 2012 with DHCP installed, but not configured (DC04)
- 2 old servers that have been unauthorized (DC01 and DC02)
- From DC03 powershell (run as administrator)
- Export-DhcpServer -ComputerName DC03 -File C:\export\dhcpexp2012.xml -Verbose
- Note, we don't want the leases
- Move the export file over to DC04 locally. The next steps can be done from DC03, but I found that it ran much faster locally then remotely.
- From DC04:
- DHCP role should already be installed, if not install it.
- Open powershell with run as administrator
- Import-DhcpServer -ComputerName DC04 -File C:\export\dhcpexp2012.xml -ServerConfigOnly -Verbose -BackupPath C:\export\backup
- This imports the server config only. No leases or scopes
- Refresh screen
- Right click DC04 and authorize the server
Now lets setup failover / replication
- These next steps are done from DC03, it will be our active server while DC04 will be the standby server
- Right click on IPv4
- Configure Failover
- Select the network - Next
- Change name if desired
- Change to mode to "Hot Standby"
- Enter a shared secret
- Next
- Close
- Refresh DHCP mmc
That's it. Now if you right click on a scope and go to properties you will find a failover tab. This will tell you if it's in failover and what the role of that particular server is.
At this point I would run netscan again and check which servers it sees handing out DHCP as well as which servers it shows as AD-authorized. In my case it shows DC03 as handing out DHCP and it shows AD authorized as DC03 and DC04.
http://www.softperfect.com/products/networkscanner/
Note: dhcploc.exe is included with the Microsoft tools on the Windows 2003 cd. (believe it's also on the 2008+ cd's). To run it I typed "dhcploc.exe mymachinesIPaddress". I took awhile to start displaying.
Friday, January 4, 2013
Installing / Upgrading Windows Server 2012 KMS Host
This guide is to install Volume Activation Services on Windows Server 2012 and then point clients to the new host. I used KMS and have not looked into Active Directory-Based Activation (which requires Windows Server 2012 AD DS schema).
First, you should read this to better understand how the process works:
http://technet.microsoft.com/en-us/library/hh831612.aspx
Confirming settings and DNS
To start with I checked my DNS to see what my VLMCS records where pointing at and found that nothing had changed yet (I had statically created them with my 2008 R2 KMS host).
This can be found under Forward Lookup Zones\your domain\_tcp it's the _VLMCS record (you may not have one depending on your past setup)
Open command prompt and navigate to C:\Windows\System32
cscript slmgr.vbs /dlv
From here we can see:
Let's add the new VLMCS record so that we can activate another 2012 Server
in the _tcp location:
I then deleted the old _VLMCS key and waited for replication.
From here on another 2012 server I issued cscript slmgr.vbs /ato which errored.
Looking in the eventvwr I could see that it was still attempting to hit the old KMS server. (event id 12288, server name is in the details).
After and ipconfig /flushdns (plus I had went to get coffee, so the time wait may have been sufficient) I ran the cscript slmgr.vbs /ato again. Now I could see it hitting the new server properly.
Now the server responds with:
Error: 0xC004F038 The software licensing service reported that the computer could not be activated. The count reported by your Key Management Service (KMS) is insufficient.
Once the KMS server hits it's 5 server count you'll be all set and it will begin giving out licenses.
Note: I found that even though the KMS server showed activated a majority of my servers where 2008 R2, but only 2 servers where 2012. The 2012 servers where giving the following:
Error: 0xC004F038 The software Licensing Service reported that the computer could not be activated. The count reported by your Key Management Service (KMS) is insufficient. Please contact your system administrator
This indicates that the KMS server hasn't hit the 5 count and activated.
2008R2 servers where activating against it without issue.
It would appear that the activation count for 2008R2 and Windows 2012 are seperate when hosted on Server 2012. After I brought up another 3 Windows 2012 servers the KMS no longer reported the count insufficient for 2012 and the 2012 servers no longer reported the 0xC004F038 error, but rather activated as expected.
First, you should read this to better understand how the process works:
http://technet.microsoft.com/en-us/library/hh831612.aspx
- Once you have your new Windows Server 2012 setup you'll need to install the new role and it's required features for Volume Activation Services.
- Once this is installed you'll find a new "VA Services" option on the Server Manage
- Right click on the server in the open Servers field and select "Volume Activation Tools".
- Click Next
- Ensure KMS is selected and the proper server is listed. You will need to be logged in as an Enterprise Administrator. Click Next
- Retrieve your KMS key from the MS VLCS login, you can find it under the relationship summary or under downloads and then clicking Keys next to Windows Server 2012. Make sure you get the one listed as KMS (not MAK) and from the proper agreement.
- Input your KMS key
- Click Yes, this uninstalls the GVLK
- Activate the product. Can be done Online or by phone.
- Click Yes to confirm
- This will return with your configuration options. Notice Licensing Status returned "Licensed"
- You are now presented with the options to for firewall exceptions and DNS (srv record).
- Click Yes to confirm
- Click Close
- You should now see in the servers window that it shows Windows Activation as "Activated"
- Right click the server again and choose the option "start performance counters"
Confirming settings and DNS
To start with I checked my DNS to see what my VLMCS records where pointing at and found that nothing had changed yet (I had statically created them with my 2008 R2 KMS host).
This can be found under Forward Lookup Zones\your domain\_tcp it's the _VLMCS record (you may not have one depending on your past setup)
Open command prompt and navigate to C:\Windows\System32
cscript slmgr.vbs /dlv
From here we can see:
- It's licensed and activated
- The partial license key
- Current count of clients (zero atm)
Let's add the new VLMCS record so that we can activate another 2012 Server
in the _tcp location:
- Other New Record
- Service Location (SRV)
- Service = _VLMCS
- Protocol = _tcp
- Port Number = 1688
- Host = fqdn of server
I then deleted the old _VLMCS key and waited for replication.
From here on another 2012 server I issued cscript slmgr.vbs /ato which errored.
Looking in the eventvwr I could see that it was still attempting to hit the old KMS server. (event id 12288, server name is in the details).
After and ipconfig /flushdns (plus I had went to get coffee, so the time wait may have been sufficient) I ran the cscript slmgr.vbs /ato again. Now I could see it hitting the new server properly.
Now the server responds with:
Error: 0xC004F038 The software licensing service reported that the computer could not be activated. The count reported by your Key Management Service (KMS) is insufficient.
Once the KMS server hits it's 5 server count you'll be all set and it will begin giving out licenses.
Note: I found that even though the KMS server showed activated a majority of my servers where 2008 R2, but only 2 servers where 2012. The 2012 servers where giving the following:
Error: 0xC004F038 The software Licensing Service reported that the computer could not be activated. The count reported by your Key Management Service (KMS) is insufficient. Please contact your system administrator
This indicates that the KMS server hasn't hit the 5 count and activated.
2008R2 servers where activating against it without issue.
It would appear that the activation count for 2008R2 and Windows 2012 are seperate when hosted on Server 2012. After I brought up another 3 Windows 2012 servers the KMS no longer reported the count insufficient for 2012 and the 2012 servers no longer reported the 0xC004F038 error, but rather activated as expected.
Monday, December 17, 2012
Citrix Receiver - There are no apps available at this time
Configuring the Netscaler Access Gateway for iOS device access we where able to get the initial setup, but then all subsequent logins to the account would result in an error:
"There are no apps available at this time"
There is a lot of info out there on this issue, but I only found a handful that where useful.
This helped me resolve the issue:
In the end, my policies look like the following which work a treat.

"There are no apps available at this time"
There is a lot of info out there on this issue, but I only found a handful that where useful.
This helped me resolve the issue:
- Set the policies so that the Receiver policy has a higher priority
- Ensure that you don't misspell Receiver! The guide I followed had the ei backwards, I read it like a robot and that cost me 2 hours of troubleshooting! Doh!
In the end, my policies look like the following which work a treat.
Netscaler / Access Gateway 401 - Unauthorized: Access is denied due to invalid credentials
Setting up the Access Gateway on a Netscaler to replace an 2010 appliance.
Found this guide to be very good for initial setup:
http://blogs.citrix.com/2012/04/10/netscaler-for-the-xendesktopxenapp-dummy/
After setup though we where still having issues with the following error:
401 - Unauthorized: Access is denied due to invalid credentials
You do not have permission to view this directory or page using the credentials that you supplied.
Looking at the Web Interface logs we found:
Event ID: 18001
A communication error occurred while attempting to contact the Access Gateway authentication service at https://xxxxxxxx/CitrixAuthService/AuthService.asmx. Check that the authentication service is running. The message reported by the underlying platform was: Unable to connect to the remote server. [Unique Log ID: 88ee87ee]
With this I was able to find Shaun Ritchie's excellent blog:
http://www.shaunritchie.co.uk/access-gateway-401-unauthorized-access-is-denied-due-to-invalid-credentials
Resolution for us was to add an entry into the hosts file pointing the FQDN to the Access Gateway virtual server internal IP address.
Note: can also be changed in DNS for your domain.
Found this guide to be very good for initial setup:
http://blogs.citrix.com/2012/04/10/netscaler-for-the-xendesktopxenapp-dummy/
After setup though we where still having issues with the following error:
401 - Unauthorized: Access is denied due to invalid credentials
You do not have permission to view this directory or page using the credentials that you supplied.
Looking at the Web Interface logs we found:
Event ID: 18001
A communication error occurred while attempting to contact the Access Gateway authentication service at https://xxxxxxxx/CitrixAuthService/AuthService.asmx. Check that the authentication service is running. The message reported by the underlying platform was: Unable to connect to the remote server. [Unique Log ID: 88ee87ee]
With this I was able to find Shaun Ritchie's excellent blog:
http://www.shaunritchie.co.uk/access-gateway-401-unauthorized-access-is-denied-due-to-invalid-credentials
Resolution for us was to add an entry into the hosts file pointing the FQDN to the Access Gateway virtual server internal IP address.
Note: can also be changed in DNS for your domain.
Friday, September 7, 2012
Lenovo ThinkPad T530 cannot install video driver - NVIDIA
Recently we purchased a new Lenovo ThinkPad T530 2392-4DU, unfortunately we had to downgrade it to Windows XP for the time being. All installed very well except the graphics driver.
When I tried to install the NVIDIA driver it would error with "NVIDIA Installer cannot continue - This graphics driver could not find compatible graphics hardware."
I tried switching to Intel integrated which wouldn't install and reported "This system does not meet the minimum requirements".
After talking to Lenovo support and for a reason unknown to me we troubshot the UEFI / SATA Controller Mode option (case of support rep not listening and going by a script) I was told that the installer package erroring was a Microsoft issue. Yeah, right. After explaining the issue again and that it wasn't a MS issue, but a Lenovo driver issue I was told that they can't help, so sorry. Long wait on hold for the manager and now it's being escalated, they'll call back next week.
Good golly. Searching around online referencing the past T520 model and I found this:
http://support.lenovo.com/en_IN/downloads/detail.page?submit=true&componentID=1345028200599&DocID=HT062424
Change it to Discrete Graphics, reboot, install NVIDIA driver.
When I tried to install the NVIDIA driver it would error with "NVIDIA Installer cannot continue - This graphics driver could not find compatible graphics hardware."
I tried switching to Intel integrated which wouldn't install and reported "This system does not meet the minimum requirements".
After talking to Lenovo support and for a reason unknown to me we troubshot the UEFI / SATA Controller Mode option (case of support rep not listening and going by a script) I was told that the installer package erroring was a Microsoft issue. Yeah, right. After explaining the issue again and that it wasn't a MS issue, but a Lenovo driver issue I was told that they can't help, so sorry. Long wait on hold for the manager and now it's being escalated, they'll call back next week.
Good golly. Searching around online referencing the past T520 model and I found this:
http://support.lenovo.com/en_IN/downloads/detail.page?submit=true&componentID=1345028200599&DocID=HT062424
Change it to Discrete Graphics, reboot, install NVIDIA driver.
Thursday, May 24, 2012
Windows 7 Trusted Sites for all users - Active Setup
On a recent deployment of laptops I needed to add a certain domain to trusted sites for all user accounts (local and domain) on a number of domain attached computers. Idealy I would use GPO to do this, but I also had to hit the local user accounts (in fact, local accounts would be used almost exclusively for these laptops).
With Windows XP we could do this by adding the proper keys to HKLM\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\Domains\etc. This would apply the domain to the trusted sites for all users on the computer (note that the site was not visible in Internet Options, but still applied), but in Windows 7 the HKLM option doesn't work anymore. Of course adding the value to HKCU functioned the same as in the past, but I can't do this for each local account all on these computers. Not to mention should I ever need to change it.
Note: There is a GPO to add trusted sites, but when it's enabled it removes the ability for the end user to then add to the list (greyed out).
After poking around I found several mentions of using Active Setup which sounded very promising. Only problem was no one (that I read) really points out how to use it.
To start, here's a great writeup of what Active Setup is: http://www.sepago.de/helge/2010/04/22/active-setup-explained/
And this wiki site briefly hits on it: http://wpkg.org/Adding_Registry_Settings
But again, neither of these really says how to use it. So, by looking at the sites above we get a brief rundown of how it works. blah blah blah by checking on user logon the values in the HKCU with the values in HKLM it knows if it's been applied and runs if it hasn't.
Cool, so if we add a value to HKLM and it hasn't be added to HKCU then it applies, yeah! Even better, we CAN manage HKLM from GPO. So, using Active Setup we can apply settings to the Local Users using Computer Configuration GPO's.
To add an Active Setup key:
All done.
User logs in and it detects that it hasn't been "installed" and runs the command adding the key. Now the user checks his trusted sites via the GUI and see's it and can even modify (to include delete the key you just added). When you want to updated the key you just make your changes to the StubPath and increment the Version value.
Now to push via GPO you just use either an adm or push with registry preferences under Computer Configuration.
With Windows XP we could do this by adding the proper keys to HKLM\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\Domains\etc. This would apply the domain to the trusted sites for all users on the computer (note that the site was not visible in Internet Options, but still applied), but in Windows 7 the HKLM option doesn't work anymore. Of course adding the value to HKCU functioned the same as in the past, but I can't do this for each local account all on these computers. Not to mention should I ever need to change it.
Note: There is a GPO to add trusted sites, but when it's enabled it removes the ability for the end user to then add to the list (greyed out).
After poking around I found several mentions of using Active Setup which sounded very promising. Only problem was no one (that I read) really points out how to use it.
To start, here's a great writeup of what Active Setup is: http://www.sepago.de/helge/2010/04/22/active-setup-explained/
And this wiki site briefly hits on it: http://wpkg.org/Adding_Registry_Settings
But again, neither of these really says how to use it. So, by looking at the sites above we get a brief rundown of how it works. blah blah blah by checking on user logon the values in the HKCU with the values in HKLM it knows if it's been applied and runs if it hasn't.
Cool, so if we add a value to HKLM and it hasn't be added to HKCU then it applies, yeah! Even better, we CAN manage HKLM from GPO. So, using Active Setup we can apply settings to the Local Users using Computer Configuration GPO's.
To add an Active Setup key:
- Navigate to regedit
- HKLM\Software\Microsoft\Active Setup\Installed Components
- Here you see the list of GUID's from other software / setups
- Add a new key, in my example I'm going to call it {newtrustedsite}. It can be called anything, but has to be unique (duh)
- Within this add a new string value named "Version"
- Give Version a value, but don't use period. Use commas instead. For instance "1,0,1"
- New string named "StubPath". Here what you want it to do. Could be an application to execute, script, other cmd line something.
- We're going to add a trusted site so my StubPath looks like this "reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\Domains\contoso.com" /v https /d 2 /t REG_DWORD /f
All done.
User logs in and it detects that it hasn't been "installed" and runs the command adding the key. Now the user checks his trusted sites via the GUI and see's it and can even modify (to include delete the key you just added). When you want to updated the key you just make your changes to the StubPath and increment the Version value.
Now to push via GPO you just use either an adm or push with registry preferences under Computer Configuration.
Friday, May 18, 2012
Windows could not parse or process unattend answer file
Had a lot of issues with cloning a Windows 7 system when using an unattend.xml
Windows could not parse or process the unattend answer file for pass [specialize]. The settings spefcified in the answer file cannot be applied. The error was detected while processing settings for component [Microsoft-Windows-Shell-Setup].
After much searching I found this which was spot on:
http://jamiebaldanza.org/2010/03/31/copyprofile-does-not-process-and-causes-windows-could-not-parse-or-process-the-unattend-answer-file-for-the-pass-specialize/
Thank you Jamie!
I had set the copyprofile paramater and had ensured that only the administrator account was present by removing the other profiles through the user control panel.
Seems that when I removed the users through the control panel it did not delete the values in the profilelist registry value.
To fix rather than making a new sysprep image I did the following:
Windows could not parse or process the unattend answer file for pass [specialize]. The settings spefcified in the answer file cannot be applied. The error was detected while processing settings for component [Microsoft-Windows-Shell-Setup].
After much searching I found this which was spot on:
http://jamiebaldanza.org/2010/03/31/copyprofile-does-not-process-and-causes-windows-could-not-parse-or-process-the-unattend-answer-file-for-the-pass-specialize/
Thank you Jamie!
I had set the copyprofile paramater and had ensured that only the administrator account was present by removing the other profiles through the user control panel.
Seems that when I removed the users through the control panel it did not delete the values in the profilelist registry value.
To fix rather than making a new sysprep image I did the following:
- Exported the wim from WDS
- Mounted my image from the wds server
- imagex /mountrw d:\
2 c:\mountedimage - Modified the system registry values
- Open Regedit
- Highlight HKEY_LOCAL_MACHINE
- File - Load Hive
- Navigate into your mounted image to Windows\System32\Config
- Select the SOFTWARE (no extension)
- Give a temp name to the hive
- Navigate into the temp hive mount and remove the obsolete keys
- Select the temp hive name
- File - Unload Hive
- Commit changes
- imagex /commit c:\mountedimage (I found that using commit and unmount in seperate command limited the amount of times I get the error that it couldn't fully unmount)
- Unmount
- imagex /unmount c:\mountedimage
- Imported the wim back into WDS as a new build
- Right click my install images group and "Add Install Image"
Wednesday, May 16, 2012
Add Driver Package to Capture Image - Windows Deployment Services 2008 R2
Using Windows Deployment Services 2008 R2 with a Capture Image of Version 6.1.7600 I have now had two instances where the network doesn't load. In both cases I've found that this is due to the network driver missing from the capture image.
I've found a lot of posts and blogs on how to add the drivers to the image and some do it the easy way, but a good number of them don't.
After booting from the capture image you attempt to enter the servername / ip address for the WDS and hit connect. This results in:
Error: The network location cannot be reached. For information about network troubleshooting, see Windows Help.
At this point if you press Shift F10 to open command prompt and type wpeutil InitializeNetwork followed by ipconfig it will result in a blank response. Running ipconfig /renew results in "The operation failed as no adapter is in the state permissible for this operation"
Add the driver to WDS:
Note: you can test the NIC prior to adding it to the image by getting the NIC drivers on a floppy or USB, PXE booting to capture, pressing Shift F10 to open cmd prompt, running drvload "path to .inf" and then issuing wpeutil InitializeNetwork.
I've found a lot of posts and blogs on how to add the drivers to the image and some do it the easy way, but a good number of them don't.
After booting from the capture image you attempt to enter the servername / ip address for the WDS and hit connect. This results in:
Error: The network location cannot be reached. For information about network troubleshooting, see Windows Help.
At this point if you press Shift F10 to open command prompt and type wpeutil InitializeNetwork followed by ipconfig it will result in a blank response. Running ipconfig /renew results in "The operation failed as no adapter is in the state permissible for this operation"
Add the driver to WDS:
- Download the NIC from the manufacturer and extract the .inf / other files.
- Place the driver files on the WDS server. In my case I put them in the following directory: C:\Drivers\Lenovo\E520\x64 and x86
- Open your WDS console
- Right click on Drivers
- Add Driver Package
- Select driver from a folder
- Navigate to the Drivers folder
- Next
- You should see it list all the drivers it finds
- Check the ones you want and press next
- Next
- It will copy the drivers
- Next
- Select or create a new driver group
- Finish
- Ensure your capture image isn't in use
- Go to Boot Images
- Select your Capture Image
- Right click and "Add Driver Packages to Image"
- Next
- "Search for Packages"
- In the results pane select the drivers you want to add
- Next
- Image will be mounted, drivers added and dismounted
- Finish
Note: you can test the NIC prior to adding it to the image by getting the NIC drivers on a floppy or USB, PXE booting to capture, pressing Shift F10 to open cmd prompt, running drvload "path to .inf" and then issuing wpeutil InitializeNetwork.
Thursday, May 3, 2012
Windows 2008 R2 View Print Server Properties
In Windows 2003 / 2008 to view the installed print drivers you could open your print server properties by simply right clicking white space in the printers windows.
With Windows 2008 R2 the process is just as easy, but since it's now different, it can be very difficult to find...

With Windows 2008 R2 the process is just as easy, but since it's now different, it can be very difficult to find...
- Open Devices and Printers
- Highlight any object under the Printers and Faxes section
- At the top of the window three new options appear next to the normal "add a device" and "add a printer"
- See What's printing
- *Print Server Properties
- Remove Device
Tuesday, February 21, 2012
Exchange 2010 SP1 ActiveSync device lockdown
The other day I began working on locking down ActiveSync so that only pre-approved devices could sync. I found that there where a lot of things available that gave portions of the solutions or hinted at parts, but none that gave the full solution (excuding one site which I didn't find until later).
I'm not going to give a full layout of all the options, but below I intend to layout the method that I'm using. From this it can easily be modified to do add rules for the needs or your organization.
ActiveSync in Exchange 2010 SP1 can control devices by allowing (default), blocking, and quarantine of the device. This is handled by identifying the device via the Device ID. The Device ID appears to differ depending on the device type, but for example with Apple's iPad the id is the Serial Number with Appl appended to the begining. So, ApplDFGGYUDVBFJ2 is what one might look like if your serial number was DFGGYUDVBFJ2. My understanding is that Andoid is not generated from the serial number.
I'm going to cover doing this through Powershell. It can also be done through the Exchange ECP via this excellent post by the exchange team (this is the one I found after I was done doing it through Powershell)
http://blogs.technet.com/b/exchange/archive/2010/11/15/3411539.aspx
Set-CASMailbox -Identity aarons -ActiveSyncAllowedDeviceIDs "ApplDFGGYUDVBFJ2","ApplDFGGYUDVBFJ3"
To set a user back to no allowed devices use the following:
Set-CASMailbox -Identity aarons -ActiveSyncAllowedDeviceIDs $Null
Note: I found that sometimes it could take awhile for a device in quarantine to generate an email stating it was in quarantine. This appeared to be because it didn't actually go to quarantine, but it was just straight out denied. In my tests this occured when the mailbox was large or contained a large number of messages.
The other part of this controlled setup that we origninally implemented was to also disable ActiveSync for all users and then explicitely enable it for users at the time of need, but if all devices go to quarantine or are blocked this no longer matters and stands in as a way to disable ActiveSync for all users without actually disabling ActiveSync (in case someone still wants to disable ActiveSync: Get-CASMailbox -identity username | Set-CASMailbox -ActiveSyncEnabled $False , you can exclude the -identity username to make it apply to all users)
I'm not going to give a full layout of all the options, but below I intend to layout the method that I'm using. From this it can easily be modified to do add rules for the needs or your organization.
ActiveSync in Exchange 2010 SP1 can control devices by allowing (default), blocking, and quarantine of the device. This is handled by identifying the device via the Device ID. The Device ID appears to differ depending on the device type, but for example with Apple's iPad the id is the Serial Number with Appl appended to the begining. So, ApplDFGGYUDVBFJ2 is what one might look like if your serial number was DFGGYUDVBFJ2. My understanding is that Andoid is not generated from the serial number.
I'm going to cover doing this through Powershell. It can also be done through the Exchange ECP via this excellent post by the exchange team (this is the one I found after I was done doing it through Powershell)
http://blogs.technet.com/b/exchange/archive/2010/11/15/3411539.aspx
- Open the Exchange Management Shell
- Get-ActiveSyncOrganizationSettings | fl DefaultAccessLevel
- This will report your current default level, most likely it's set to Allow
- Set-ActiveSyncOrganizationSettings -DefaultAccessLevel Quarantine -AdminMailRecipients admin@didyourestart.com
- Here we are setting the default level to Quarantine and then specifying an email address to be notified when a device is quarantined. You will want this so that you can see what the DeviceID is of new devices (makes for easier adding of devices for allow access)
- You'll now find that if you try to connect your iPad it will be quarantined. Quarantined devices can be found via the Exchange ECP as described in the link above to blogs.technet.com or via Powershell
- Get-ActiveSyncDevice | where {$_.deviceaccessstate -eq 'Quarantined'} | ft DistinguishedName
- This can also be used to find the DeviceID. On iPad this results in simular to following
- CN=iPad§ApplDFGGYUDVBFJ2,CN=ExchangeActiveSyncDevices,.....
- Next step is to allow this device access for the user
- Set-CASMailbox -Identity username -ActiveSyncAllowedDeviceIDs "ApplDFGGYUDVBFJ2 "
- You can view the list of allowed devices at anytime by using the following
- Get-CASMailbox -Identity aarons | fl ActiveSyncAllowedDeviceIDs
Set-CASMailbox -Identity aarons -ActiveSyncAllowedDeviceIDs "ApplDFGGYUDVBFJ2","ApplDFGGYUDVBFJ3"
To set a user back to no allowed devices use the following:
Set-CASMailbox -Identity aarons -ActiveSyncAllowedDeviceIDs $Null
Note: I found that sometimes it could take awhile for a device in quarantine to generate an email stating it was in quarantine. This appeared to be because it didn't actually go to quarantine, but it was just straight out denied. In my tests this occured when the mailbox was large or contained a large number of messages.
The other part of this controlled setup that we origninally implemented was to also disable ActiveSync for all users and then explicitely enable it for users at the time of need, but if all devices go to quarantine or are blocked this no longer matters and stands in as a way to disable ActiveSync for all users without actually disabling ActiveSync (in case someone still wants to disable ActiveSync: Get-CASMailbox -identity username | Set-CASMailbox -ActiveSyncEnabled $False , you can exclude the -identity username to make it apply to all users)
Subscribe to:
Posts (Atom)