Let's see if we can illuminate what Hamilton said about the characteristic function in his article on the Paths of Light and the Planets. First he assumed that characteristic function was invariant for changes in the path and this allowed him to show that the path was a straight line. One can add that since the characteristic function is independent of the path it is a "state" function and depends only on the initial and final points of the motion. To understand the derivation of the differential equation for the characteristic function let's look at the distance of a plane from the origin. The distance can be expressed as a function of the distances of the points where the plane intercepts the coordinate axes.
A translation of the origin doesn't change the distance of the point from the plane so we know the distance of the point from the in terms of its distances from the plane measured parallel to the axes. If we keep the direction defining the plane fixed and allow the distance of the plane to vary we get its change in distance in terms of the partial derivatives and this allows us to derive a differential equation for the characteristic function. Using the gradient allows us to simplify the equation some.
Hamilton's characteristic function allows us to determine a set of paths for the motion of particles starting from varying positions and set up surfaces for which the characteristic function is constant. The paths and the surfaces are normal to each other. The straight lines are a special case but it works for light when the index of refraction is constant everywhere. If we have a set of parallel paths the corresponding surfaces will be planes. If they radiate from a point the surfaces will be spherical.
Supplemental (Aug 21): I found the equation for the normal distance of a point from a plane in the 1980's while looking for an alternative to the usual way of measuring errors vertically in least squares. Something clicked when I was trying to illuminate what Hamilton was doing.