There seems to be a lot of confusion and myths as far as the /3GB and /PAE Switch being used on the Windows server platform operating systems.  I am trying to clarify by the below explanation

You should not use /3GB with /PAE ( >4GB of physical memory) PAE switch if we have more than 4GB of RAM
The /PAE parameter enables Physical Address Extension (PAE). This parameter directs the system to load the PAE version of the Windows kernel. This will allow a 32Bit Operating system to read beyond the 4 Gigabyte limit. (PAE kernel file: Ntkrnlpa.exe)/3GB support, just like AWE support, is included in 'lower level' versions of the operating system so that application, utility and adapter vendors can develop and test their products without the need to buy more expensive versions of the operating system
The virtual address space of processes and applications is still limited to 2 GB unless the /3GB switch is used in the Boot.ini file. When the physical RAM in the system exceeds 16 GB and the /3GB switch is used, the operating system will ignore the additional RAM until the /3GB switch is removed. This is because of the increased size of the kernel required to support more Page Table Entries. The assumption is made that the administrator would rather not lose the /3GB functionality silently and automatically; therefore, this requires the administrator to explicitly change this setting.
The /3GB switch allocates 3 GB of virtual address space to an application that uses IMAGE_FILE_LARGE_ADDRESS_AWARE in the process header. This switch allows applications to address 1 GB of additional virtual address space above 2 GB.
The virtual address space of processes and applications is still limited to 2 GB, unless the /3GB switch is used in the Boot.ini file.
The most common problem which we face with /3 GB and /PAE is the issue with allocation of Sys PTEs and OS suffering in performance when high demand for memory is placed.

1. /3 GB switch that was implemented when windows did not support more than 4 GB of RAM. What the 3GB switch did was to give memory intensive applications more virtual address space, ensuring that it could have more files in the memory at any given time. This effectively reduced the number of pages being paged out to the disk, and ensured faster performance of database programs. The fallout was that the kernel space reduced to 1GB, this effectively reduced the memory allocation of sysptes.

2. /PAE switch ensured that the OS is able to see more than 4 GB of RAM. An AWE aware application can take advantage of this and put its pages in the in the upper 4GB of space specified to it. The Downside of this however is that with this, in order to address a page in the memory, the OS needs 2 sysptes instead of 1. So this effectively halves the number of available sysptes.
The virtual address space was designed to be 4 GB in total in the windows architecture which was assumed to be a plethora of space. Later when more and more powerful application stated to come into picture, the virtual address space limit per process was still maintained at 2 GB. On a server with more than 4 GB of RAM we can enable PAE switch and then the application process now starts using 2 GB virtual address space and then with the help of AWE and PAE starts address more physical RAM to max of 64 GB. But onto server’s which have only 4GB or less amount of RAM will limit the virtual address space of the application process to 2 GB.

Now in cases like an exchange server or SQL server with 4 GB RAM, we need the application to gain more resource than the OS itself because the OS is not doing any resource intensive work but the application is doing it. So in those situations we do want the OS to waste the virtual address space which it is not using. Hence we enable the /3GB switch and give the application 3 GB space. During accessing the virtual memory on consistence basis there is lot of fragmentations which happen. Increasing the space to 3 GB, the applications now can leverage allocating larger blocks of memory to itself keeping relative data in one single large block. This enhances the application performances to a great extent. But the caveat to increasing the user space to 3 GB is, if in case the OS needs more than one GB address space it will never get it. So then the /USERVA comes into picture and then we can fine tune and revert back few MBs to the OS from the 3 GB address space allocated to the applications.
On server which use the PAE switch should ideally disable the /3GB because the moment we start using the PAE the OS needs more resources now, and application will start allocation the RAM directly with help of AWE configuration. So when application needs more than 2 GB space it moves to AWE memory blocks and hence the performance remains best.

The conflict is when we use the /3GB with /PAE simultaneously. When we are using PAE the OS needs to manage lot of resources which needs more memory than 1 GB. So certain operations have performance issue and then fail intermittently. To come up with this limitation of address space infinitely the next version of windows that is x64 bit has 16TB of virtual address space and this is virtually inexhaustible.
A program that requests 3 GB of memory is more likely to be able to have more of its memory remains in physical memory rather than be paged out. This increases the performance of programs that are capable of using the /3GB switch.  

The theoretical 64-bit virtual address space is 16 exabytes (18,446,744,073,709,551,616 bytes,
or approximately 17.2 billion gigabytes). Unlike on the x86 where the default address space is
divided in two parts (half for a process and half for the system), the 64-bit address is divided
into a number of different sized regions whose components match conceptually the portions
of user, system, and session space. The various sizes of these regions, listed in Table below, represent
current implementation limits that could easily be extended in future releases. (The
sizes on the 32-bit x86 platform are included for comparison purposes.) Clearly, 64-bits provides
a tremendous leap in terms of address space sizes.

Adding to the above
IMAGE_FILE_LARGE_ADDRESS_AWARE flag that is mention below, in case you really want to check if an image was built to be large address aware, this how you can check it in the image header.

When the image is being built, and at the link phase, you specify the /LARGEADDRESSAWARE qualifier to the image linker, and that’s how you get this FLAG set in the image header. When the process is created and Image Loader starts loading the images into the address space, he would examine the image header and setup the address space for that image accordingly. Say you have booted the OS with 3GB switch, and the image is not LARGEADDRESSAWARE then it would be restricted back to 2GB of user address space (and 1GB of Virtual address are not going to be usable by that app anymore).

This is how you can find if the image is LARGEADDRESSAWARE or not. Dump the headers of any image using the linker (you get link.exe from SDK/DDK)

C:\>link –dump –headers c:\windows\system32\lsass.exe
Dump of file c:\windows\system32\lsass.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
             14C machine (x86)
               3 number of sections
        41107B4D time date stamp Wed Aug 04 11:29:41 2004
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             12F characteristics
                   Relocations stripped
                   Executable
                   Line numbers stripped
                   Symbols stripped
                   Application can handle large (>2GB) addresses
                   32 bit word machine