Running X11 with Fast User Switching

[Malcolm Fitzgerald]

X11 wont' let me fire it up if I have left it running and switched to  
another user. What can I do to change this?

This is what it spits into console:

XFree86 Version 4.4.0 / X Window System
(protocol Version 11, revision 0, vendor release 6600)
[DRI] screen 0 installation complete
Screen 0 added: 1280x800 @ (0,0)
Could not init font path element /usr/X11R6/lib/X11/fonts/CID/,  
removing from list!
Xlib: connection to ":0.0" refused by server
Xlib: No protocol specified

xterm Xt error: Can't open display: :0.0
Xlib: connection to ":0.0" refused by server
Xlib: No protocol specified

can't open default display
Quitting XDarwin...