The trick to small layout design is to allow as much action as possible without clutter or cramming. You want trains to move. Scenery has to be added judiciously. You don’t want to pack things tightly. A good blend of track, structures, signals and natural things makes for a pleasant layout.
Know your trains. Know what you will run on the small layout. Understand that the value of compactness has a pay-off. In gaining action in a small space, you lose the ability to run larger trains. Shorter locomotives and cars thrive on a small pike as much as a large one. Even if they can make the curves and tolerances, large trains look awkward. Overhanging passenger cars and long freight cars detract from the appeal of running a train.
Several things affect the design of the track plan. Will it be realistic, whimsical, classic “tinscale” or something else? Do you want to run operating cars and accessories on it? How important is scenery? Do you want your scenery to look realistic, “tin-litho” or toy-like? Questions like these lead to developing the layout that best suits your desires.
Operating systems are less of an issue on a compact railway. The electronic systems like TMCC, DCS and DCC hit their stride with larger layouts. Much of their value is wasted on a small pike, where everything is pretty much in reach. Indeed, manual accessories and switches can be used comfortably on a very small layout. Transformer control is more than adequate. Likewise, wiring is much easier.
Understand that different brands and styles of track have different track geometry. Pick track that fits your area nicely.
Last, what is in your budget? You can make a small layout that is within your means. You can also add to it gradually.
The main rule is that it is YOUR railroad so make it the way you want it.