Irregular packing algorithms are what you're looking for. I know they exist, and are non-trivial. In 2d, however, a manual method can be used.I have yet to see a good generic algorithm for placing stars in free-form 3D. It's simple to randomize positions starting from a fixed origin, or within a large predefined block of space. I would like to procedurally assign a randomized volume to each star system and have a way to generate a neighbor based on volume and directional limits. Just a thought.
There is program for use with 2300 and Starfire, but I don't recall the name. Worse, the Starfire Forums are down, too.