In the etc directory under your Netbeans-Home, edit the file netbeans.conf file.
-Xms and -Xmx should be increased to the values that allow your program to compile.
Here are the instructions in netbeans.conf:
# Note that default -Xmx and -XX:MaxPermSize are selected for you automatically.
# You can find these values in var/log/messages.log file in your userdir.
# The automatically selected value can be overridden by specifying -J-Xmx or
# -J-XX:MaxPermSize= here or on the command line.
Put the values in the netbeans_default_options string. Here is mine (remove linebreaks, added for readability):
netbeans_default_options="-J-client -J-Xss2m -J-Xms32m -J-XX:PermSize=32m
-J-Dapple.laf.useScreenMenuBar=true -J-Dapple.awt.graphics.UseQuartz=true
-J-Dsun.java2d.noddraw=true -J-Dsun.java2d.dpiaware=true
-J-Dsun.zip.disableMemoryMapping=true -J-Dsun.awt.disableMixing=true
-J-Dswing.aatext=true -J-Dawt.useSystemAAFontSettings=lcd --laf Nimbus"
EDIT: -J-Xms sets the minimum Java heap size, -J-Xmx sets the maximum Java heap size.