I received a very interesting bug that’s been plaguing our product for a long time now. The problem is that intermittently our WinForm client users would view an image using the WebBrowser Control that would cause the program to crash. Listing this error:

Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

This problem happen when our users attempted to use a SVG Viewer or Tiff Viewer ActiveX object. (Which worked perfectly in IE outside of our app) and this did not occur on every machine. Some machines would work, others would not. We were finally able to narrow it down to those using MS Vista’s 32 bit operating system.

After loads of research I was able to narrow it down to Vistas Data Execution Prevention (DEP).

Snippet from Wikipedia about this:

Data Execution Prevention (DEP) is a security feature that is intended to prevent an application or service from executing code from a non-executable memory region. This helps prevent certain exploits that store code via a buffer overflow, for example. DEP runs in two modes: hardware-enforced DEP for CPUs that can mark memory pages as nonexecutable, and software-enforced DEP with a limited prevention for CPUs that do not have hardware support. Software-enforced DEP does not protect from execution of code in data pages, but instead from another type of attack (SEH overwrite).

DEP was introduced in Windows XP Service Pack 2 and is included in Windows XP Tablet PC Edition 2005, Windows Server 2003Service Pack 1 and later,[1] Windows Vista, and Windows Server 2008.

Software configuration

So how does one configure DEP. There are multiple ways of configuring this. Boot.ini (XP and VISTA) ,command line and using Microsoft’s ACT 5.0. By far the easiest method is to use the command line.

1)      Find your Command prompt and right click and select “run as administrator”

2)      Then type “bcdedit.exe /set {current} nx XXXXXXX”

a.       Where XXXXXXX can be the following:

OptIn ( 2 ): This setting is the default configuration for Windows XP. On systems with processors that can implement hardware-enforced DEP, DEP is enabled by default for limited system binaries and programs that "opt-in." With this option, only Windows system binaries are covered by DEP by default.

OptOut( 3 ): This setting is the default configuration for Windows 2003 SP1. DEP is enabled by default for all processes. A list of specific programs that should not have DEP applied can be entered using the System dialog box in Control Panel. Network administrators can use the Application Compatibility Toolkit to "opt-out" one or more programs from DEP protection. System compatibility fixes, or shims, for DEP do take effect. Also note that Windows silently disables DEP for certain executables, such as those packaged with ASPack. [5]

AlwaysOn( 1 ): This setting provides full DEP coverage for the whole system. All processes always run with DEP applied. The exceptions list to exempt specific programs from DEP protection is not available. System compatibility fixes for DEP do not take effect. Programs that have been opted-out by using the Application Compatibility Toolkit run with DEP applied.

AlwaysOff( 0 ): This setting does not provide any DEP coverage for any part of the system, regardless of hardware DEP support. (except in Windows Vista Ultimate)

So your commandline might look something like this:

“ bcdedit.exe /set {current} nx AlwaysOff”

Once you are done with this you will need to restart your computer.

To check the status of your DEP Policy you can run this command:

“wmic os get dataexecutionprevention_supportpolicy” This will return a number value.  See values above to see what numbers mean.

Now this is how we defined what the problem was. But turning off DEP entirely isn’t a viable solution since that will expose your computer to many evil virus and hacking code.

Vista provides an interface to select specific programs to be marked as DEP non compliance. You can get to it by going to:

1)      Start

2)      Right Click “My Computer” à Properties

3)      Advanced System Settings

4)      Advanced Tab

5)      Under Performance select “Settings”

6)      Select the tab “Data Execution Prevention”

Here is where you are able to add DEP for all programs and services except those I select. This will set your DEP Policy to 3 (OptOut). You can then select the exe programs that enable you to mark that program as DEP noncompliant.

Well that might work for other people, but it didn’t work for me on my machine.  Everytime I tried to select my program’s exe, I got the message “This program must run with data execution protection(DEP) enabled. You cannot turn off DEP for this program.”

Later I found a website that lists a NXCOMPAT and the C# compiler.

Turns out that you can add this switch to your postbuild event of your project and it will mark your compiled project as DEP non compliant.

1) Right click on project in Visual Studio

2) Properties

3) Build Events

4) Edit Post Build ...

REM Mark project as DEP Noncompliant

call "$(DevEnvDir)..\..\VC\bin\vcvars32.bat"

call "$(DevEnvDir)..\..\VC\bin\editbin.exe" /NXCOMPAT:NO "$(TargetPath)"

Another way to do this is to open up the Visual Studio command prompt, browse to your exe location and type:

editbin.exe /NXCOMPAT:NO YourProgram.exe

That’s it, now our product will compile and it fixes many crash log errors we were receiving previously.