Windows Server 2012, VMware ESXi, and Strange Permissions Errors – Removable Storage Auditing what?

I’ve been getting ready to roll out Windows Server 2012 to be used for new production hosts, and one of the first machines I tried to deploy was a web server. I created my VM, installed Windows 2012, got IIS all set up. This web site was to be used for uploading and serving files, so I added a data disk and put the web site on this drive. The web app runs on Perl so I installed that, configured it, and no matter what I kept getting 500.19 errors. I don’t set up Perl all that often on IIS, and I thought for sure that it must have been some quirk in the process that I was missing.

Instead it turned out to be an unholy combination of factors with the way that VMware presents disk drives to Windows and how Windows 8 / Server 2012 handles removable drives. It effects things other than IIS too. The symptoms are strange, the errors are unhelpful, and the fixes aren’t obvious.

What VMware is Doing

Starting with (I think)  ESXi 5, VMware presents whatever devices it can to Windows as hot-pluggable. So the network adapter and the disk drives show up in that little menu in the tray where you can tell Windows to eject a device.

What Server 2012 is Doing

Windows 8 and Windows 2012 have some weird auditing setting pertaining to removable drives. After all the researching I’m still not sure how auditing removable drive access changes whether the access is successful or not (no Heisenberg jokes please). But somehow this is what’s going on.

Changing the Audit Policy

Most of the articles and discussions I’ve seen focus on changing the auditing policy. If you’re not using Core, you’d open gpedit.msc to look at the local policy, and then navigate to:

Computer Configuration\Windows Settings\Security Settings\Advanced Audit Policy Configuration\System Audit Policies\Object Access

The setting is Audit Removable Storage  and it’s set to a very innocuous looking Not Configured by default. The proposed fix is to Enable the policy but leave the Success and Failure checkboxes unchecked. Effectively this sets the value to “No Auditing.”

I tried this method and it worked for the IIS issue.

You can also set this through Group Policy, but you have to do it through a Windows 8 or Server 2012 Group Policy Management Console, because the setting doesn’t exist in the previous clients!

Enter WSUS 4.0

I was having issues updating Windows 2012 and Windows 8 using WSUS 3.0 SP2. After reviewing the options, I felt the best way to go was to just deploy WSUS 4.0 in Windows 2012. I followed the migration guide but I was getting permissions errors with the WSUS content directory stored on a drive other than C:. I tested the above policy fix and this did not work for WSUS.

Better Living Through Auditpol

At this point I wasn’t fully aware of why these things were happening, and I was searching for the WSUS error, but I stumbled across this post from Happy Sysadmin that shed more light on the whole situation. WSUS wasn’t mentioned but this post has 2 fixes which I didn’t see elsewhere.

Fixing it in VMware

One way to solve this whole mess is to disable hotplug in VMware. If Windows doesn’t see the devices as hotpluggable, it won’t apply the removable storage auditing. I tried this, and it worked great. I personally like the hotplug feature though.

The Windows Solution

My favorite fix was the suggestion to use this command:

auditpol /set /subcategory:"Removable Storage" /failure:disable

This worked for WSUS as well. The interesting thing is that seems like this would do the same thing as setting the policy to No Auditing, but it simply doesn’t. After running this command, the policy still shows up as Not Configured, and the effect does seems to be different.

Errors Caused by Removable Storage Auditing

The main reason I’m writing this even though there are other articles out there is that I want to compile a list of issues that are attributable to this behavior, and to be clear that the most proposed fix did not seem to work for me on one of those incidents. I suspect the VMware fix will always work and I’m almost as sure that the auditpol command will always work.

1. IIS 500.19 errors when site is located on non-system drive

For me, all of the fixes worked.

2. WSUS 4.0 cannot successfully download updates

The way this manifested itself for me was that WSUS would act like it was downloading updates from Microsoft but no progress ever happened. When I ran wsusutil to check the integrity of the downloads, it wanted to re-download all of the updates. Still no progress. Reluctantly, I deleted everything on the volume. Still no progress. The manual policy setting did not work for me, but I was using Core and had to set it indirectly via Group Policy. Maybe I messed something up? But the auditpol command worked and the VMware fix worked too.

3. Access denied to disk share

I didn’t personally experience this one, but it was the main subject of the Happy Sysadmin article I linked above. Obviously the 2 workarounds listed there worked.

Send me More!

I want to hear more about similar issues. Contact me or leave a comment, and I’ll post about your experiences. Include error messages and numbers, steps to repeat when possible.

Thanks again to Happy Sysadmin.

 

Comments

  1. Sysadmin 3.14 says:

    We have seen similar issues with (1) SharePoint 2013 having access denied errors when writing to log files locally and (2) SQL Server 2012 running the database engine service under a domain account (access denied accessing database files on service start with domain account, fine running as LocalSystem). Disabling hot plug devices in ESXi 5.1 resolved both issues.

    • Interesting, thanks for this. I’m still on SharePoint 2010 in my installation, but if/when we move to 2013 I’ll definitely be looking out for this. I’m running SQL 2012 but not on Windows 2012 yet, so that will also come in handy.

  2. Thank you! I ran into the very same problems with IIS 500.19 when putting wwwroot on the second disc of my VM. I found the solution couple of weeks ago on a page I cannot find anymore. But now I have something to forward to our VMWare guys…

  3. Eric Brousseau says:

    Thanks a lot! I had similar issues with SQL Server 2012 / Windows Server 2012 / ESXi 5.5 when trying to access database files located on anything but the C:\ drive if the SQL Server service wasn’t running under the local system account. Setting the policy solved the issue. Great find!

Speak Your Mind

*