However I assume everything has already been tried and kept or rejected for good reason, and I don't see any kayak designs with them but still wonder if hull rails would be advantageous.
Gerald is correct in that it is not going to happen in a human-powered craft except an extremely specialized hull with an extremely powerful human engine, and IF then,...only for an extremely brief period(seconds).
The power-to-weight ratio is just wrong ( For a discussion of an optimum displacement/planing "transitional" hull, see "High Speed Sailing, by Frank Bethwaite). Frank gives minimum power to weight figures for planing, and it's not doable in any practical way by we human engines. (Sails give Frank enough power to work with, and he does it very well!)
[Edit - planing trivia):
The only readily-observable demonstration of human-powered planing on the water's surface is the example of kids riding on 'skim-boards' in the shallows at the beach(or similar shallow-water location).
The board(planing surface) is dropped (and then jumped upon) by the already-up-to-speed youngster, and "planes" with it's usually quite-lightweight load for a matter of a few seconds, until the speed drops below what is needed to create the "dynamic" lift.
This type of planing takes place in and is aided by the "ground effect" condition, due to the proximity of the bottom in the shallow water.
Nevertheless, it is definitely "planing" rather than displacement operation, as the board certainly does not displace sufficient water to support the weight of itself and its rider.
"Rails", and "lapstrake":
Rails can be useful as protective rub-rails, and as external hull stiffeners(stringers). A keel-like rail or skeg that increases lateral plane, especially toward the ends of the hull, can act to counter yawing forces from side-to-side power application(paddling, as opposed to rowing) or other yaw-inducing forces(greater windage fore or aft in a cross-wind, for example).
"Lapped"-strakes provide stiffness in a lightweight hull, because each lap is in effect a full-length "stringer" which is thicker than the "skin" between the laps.
Other than these functions, the rails add weight and the rails and lap-strakes add building complexity, and both increase skin(surface)drag, ... none of which aids stability or speed.
Sharp corners(chines) vs. round:
Look at the simplest examples to understand the principles. A square shape shifts the center of buoyancy out-board when a roll begins, to some degree aiding stability. A round form(think log-rolling) does not. (See the B-B method below, for a visual aid).
Either chine-shape is only the begining, however, and the "flare"(concavity), or "flam"(convexity) of the hull which was previously above the waterline comes into play here, and is a whole 'nother discussion.
Here is a method useful to hands-on designers, as opposed to those who prefer to let someone-else's computerized design program provide "answers" to them:
To see what a given form will do when rolled, make a raised outline of a given section, on your slightly-inclined drawing board ( there are flexible rulers you can use for this, or simply cut the shape out of a thick-enough cardboard to contain the "water-molecules
, and fill it to the waterline with marbles or B-B's. "Roll" the section, and arrange the B-B's so they are level with the original water surface. This will let you see how an immersed section (of the same displacement) is now shaped, and will give you an idea of how the center-of-buoyancy has shifted with the tilt. ( AFAIK, this "B-B's of displacement" method of determining immersed section shape is original with myself. I have an extensive design library, and have not seen it mentioned elsewhere.)
The actual C. of B. can be determined by finding the C. of G. of a cardboard cut-out of the new immersed shape. When the C.of B. is outside the C. of G. (of the boat and load) the influence willl be to counter the roll, ... i.e., a stabilizing force.
For a better idea of the stability of the whole hull, this procedure needs to be repeated for a greater number(or all) of the hull station-sections.
"Already been tried":
Yes, nearly everything. "Advances", at this late stage of development are nearly all tied to new materials and methods, ...with few really "new" ideas left to discover regarding basic design.
Look to the extreme examples for the limits. A racing shell shape is currently the fastest "displacement" human-powered craft. It has a length and shape determined to be the best compromise for minimizing wave-making and surface drags. It is "dynamically" stabilized by the sculler via the oars. The boat is completely unstable without the oars(sculls) in the water (back to log-rolling again).
Anything done to stabilize this "optimum" hull using "form stability" (which will work equally well at rest or underway,) rather than dynamic stability, has a drag cost, and reduces speed.
The best (least drag-costly) means of stabilizing such a hull is the use of "flying" outriggers - pontoons or floats which are just above the surface of the water, and do not become effective (and thus create no water-drag) unless(untill!) the boat begins to roll.
BUT!, ...these are generally not suitable in a boat where the oar-or-paddle stroke would contact the outrigger!
Form stability: The further "out" to the sides the center of buoyancy can be moved, the more roll resistance it can provide. A catamaran is the prime example. BUT! (again
, ... there is no catamaran (or other multi-hull) configuration which can have as little drag as an optimized monohull (physics). Also, a cat is not best suited to oars or a paddle, because of reach requirements, and increased yawing when alternate side power(paddling) is used.
So we have one of those unavoidable "compromises" met with in boat design. When using form stabilization, other-than outriggers , a wider boat (for a given length) is a more stable boat, but is also a slower boat.
So, the best practical compromise between stability and speed for an oarsman or paddler, as determined by thousands of years of trial and error, is a hull which is form-stabilized by it's wider-than-a-log shape, but still as "tippy" as the user is willing to accept.
Users have differing abilities or fears, so the range of "acceptable" stabilities is wide.
Is there a bottom line?
For the non-designer, trying out a number of different boats will soon give one a seat-of-the-pants understanding of what his/her personal requirements are, for a boat in which one can feel safe.
The same is required for the designer, to give a feel-basis for what is "enough" stability, after which he can play with shapes which provide it, while best meeting his other design goals (speed, etc). Success in this activity assumes quite a bit of study
on the would-be designer's part, of texts in which the tremendous amount of "already done" experimentation, and its results, and lessons-learned, are described.