I go every year in late October. Spend at least one evening with a guide (Bill Wetzel) which is worth it just for the access alone.
Unless you've been there most won't believe the acre or so of schools of bass that move across the lighthouse. I tend to stay away from the point usually, and avoid weekends. To crowded and dangerous with the weekend warriors slinging plugs without looking around them first, or worse, walk right up behind you when you are about to cast.
I split my time between surf and my brothers 18' hydro cc.