Here is a summary of what I know, with @RichardStanley's remark taken into account. It was shown independently by Polo and by Woodcock that the incidence algebra of a graded poset is Koszul iff each open interval of the poset is Cohen-Macaulay (over the field in question). See P. Polo. On Cohen-Macaulay posets, Koszul algebras and certain modules associated to Schubert varieties. Bull. London Math. Soc., 27(5):425–434, 1995 and D. Woodcock. Cohen-Macaulay complexes and Koszul rings. J. London Math. Soc. (2), 57(2):398–410, 1998.
A semimodular lattice has Cohen-Macaulay open intervals (as pointed out by @RichardStanley) and so has a Koszul incidence algebra. This includes modular lattices.
Being modular is not necessary for being quadratic. The issue is the Hasse diagram need not contain a pentagon for a non-modular lattice. The pentagon is just a sublattice but it is not "convexly" embedded.
A fairly general class of graded posets with quadratic incidence algebras (and maybe they are exactly the class) are those which are what is sometimes called strongly connected, meaning that the order complex of each closed interval is a chamber complex (meaning any two facets are connected by a gallery as people study them in the theory of buildings). This means that any two maximal chains with the same top and bottom can be connected by a sequence of moves where you change only one element of the chain at each step. Semimodular lattices are strongly conencted, for example. We prove in Lemma 7.4 of https://www.ams.org/books/memo/1345/memo1345.pdf (you can also find it on ArXiv) that the incidence algebra of a strongly connected poset is quadratic. We also observe the quadratic dual is defined simply by reversing the Hasse diagram and taking the relation which is the sum of all paths of length 2. This is the quadratic perp but it is a nicer description in this case.
We show that for geometric lattices associated to oriented matroids (including central hyperplane arrangements), the Koszul dual of the incidence algebra is the monoid algebra of the monoid of covectors of the oriented matroid. But the proof involves a nontrivial topological argument to show that this monoid has a quiver presentation matching the quadratic perp. We prove a similar thing for intersection semilattices of affine hyperplane arrangements and affine oriented matroids, as well as some other CW complexes that have a left regular band multiplication on its face poset including finite CAT(0) cube complexes. Again the incidence algebra of a certain strongly connected meet semilattice is Koszul dual to the semigroup algebra of the left regular band of the CW complex.
I don't know of a general combinatorial description of the Koszul dual.