Sounds more like a problem with your scene file.
Either it´s broken or way to large.
How large is it ? (MB)
Did you maybe import stuff from a CAD software ?
Maybe an object or vertex that is to far away from the scene root ?
Also look at your Taskmanager to see how much RAM Gmax uses.
If you are using a non modified XP32 Max will usually crash at
~2GB Memory use.
If you are using XP32 no matter how much RAM you have it will
only allocate 2GB of RAM to a certain Application anyway.
You can extend this to 3GB by using the 3GB Switch.
http://www.vfxpedia.com/index.php?title=FAQ/3GB_Switch(not necessary if you are on a 64 bit OS)
You can also clear unused memory in 3dsmax
(not sure if Gmax supports maxscript)
with a simple command
gc()
Just type it into your maxscript listener and hit enter.
Usually building realtime models shouldn´t blowup a scene that
large to run out of memory. If you suspect something wrong
with your scenefile, you should create a new file an merge
object by object into that one.