Since your question is somewhat open-ended ("what additional information is needed?"), it appears difficult to provide a complete answer. However, at least one flavor of answer may be found in David Treumann's thesis:
D Treumann, Exit paths and constructible stacks, Compositio Mathematica, Vol 145, No. 06, pp 1504- 1532 (2009)
The main result 2-categorifies a theorem of MacPherson, which states that the category of perverse sheaves over a (topologically) stratified space $X$ is equivalent to the category of set-valued functors from the exit path category of $X$.
The exit path category of $X$ has all points of $X$ as objects, and morphisms are (homotopy classes of) continuous paths which we allow to ascend up to higher strata, but never descend to lower ones. Asking that all strata be contractible does not, in general, allow one to pass from the exit path category to the poset of strata without losing structure. On the other hand, if your stratification of $X$ arises from a regular CW decomposition, then the associated exit path category retracts onto the poset of cells.
Stringing all of this together, one set of sufficient conditions which guarantee that the category of perverse sheaves is recoverable from the poset of strata, is that the stratification comes from a regular CW decomposition of $X$.
(I've also described a purely combinatorial 2-categorified version of the dual entrance path category for regular CW complexes in Sec 4.1 of this preprint).
