-1
$\begingroup$

This is a question from high-frequency trading (HFT). A market maker sends transaction requests to the exchange's server via a certain number of gateways. At these gateways the requests incur some random delay, but these delays are not systematic, they vary from gateway to gateway, and they vary over time. The delays might correlate with quantities such as traded volume, or maybe others, but that's an unknown. The problem is to find an optimal gateway selection strategy to minimize some delay metric (max or average etc.). Now, the market maker has sent me a large amount of delay data (essentially a long list of gateway number, time of day, transaction delay) and asked me to find that optimal strategy. Problem is, the data he's sent me is already the result of some degree of optimization, and he wants me to improve the strategy. Now I'm not sure this is at all possible. Given that the data is the result of some non-ergodic selection strategy, it is biased, and this bias is clearly visible as the number of transactions per hour sent to each gateway varies way more than what would be expected if the gateway selection was purely random. The data being biased, I'm not sure I can do anything with it. In my view, to come up with a strategy, I would need the delay information at all points in time, and not only at those points in time when the market maker actually sends a transaction request. I'm asking the experts here for their view on this. Thanks in advance.

PS: Maybe if I formulate the problem more mathematically it's easier to understand. Suppose I have two sets of delay measurements: the measurements in the first set were taken at completely random points in time, while the measurements of the second set were taken at points in time that predicted shorter delays. Both sets come with a small number of measured covariates. While the delay distribution resulting from the first set, by the law of large numbers, converges to the true delay distribution, the delay distribution inferred from the second set does not. The question is this: based on the second set of measurements alone, is it - theoretically - possible to derive a predictor that correlates even better with shorter delays?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I don't understand the specifics of the gateway problem well enough to comment, but on the broader question of "how do I perform inference or optimization when I have observational data, and it's probably biased?", there is a large statistical literature that attempts to wrestle with that question. (This situation occurs frequently in economics, when the only data available consists of natural experiments.)

Here's two possible starting points:

  • If you want to handle the bias question directly, you might look in to Propensity Score Matching.
  • On the other hand, given all the time series data, you may be better served by investigating Synthetic Control.

Of the two, I'd suspect that synthetic control might be more useful.

I think I'm assuming that you can break the problem into two parts: (A) handle the biases in the data to build a model, and then (B) optimize your gateway selection given the model. It's possible that (B) may be hard to think about until you've tackled (A).

(And please feel free to reach out to me directly if it would be helpful.)

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Thanks a lot for the answer Bill. Appreciated! I'm not sure I explained my problem clearly enough, so I added an alternative formulation. Maybe you can tell me if you think the two solutions you suggested could still be used. $\endgroup$
    – sakuragaoka2001
    Commented Nov 18, 2022 at 19:48
  • 1
    $\begingroup$ Thank you for the extra explanation, @sakuragaoka2001 ! That was very clarifying for me. A few thoughts: first, I kind of suspect that your question might find a more responsive audience amongst statisticians than mathematicians (e.g., consider Cross Validated / stats.stackexchange.com ). Second, it sounds like you have a biased sample of data and wish to unbias it. Debiasing is, in general, very possible to do. But the success is going to depend on the particular data-- e.g., if you are completely missing data from certain gateways for long stretches of time, and you have no... $\endgroup$
    – Bill Bradley
    Commented Nov 20, 2022 at 13:06
  • 1
    $\begingroup$ ... pre-existing model, then it's hard to see how you could do anything. But if you have at least a little data across gateways, even if it's already non-uniformly sampled, then there are a lot of tricks to play. One trick is "importance sampling"; you could, in some sense, use it "backwards" to unbias your distribution. If you suspect correlation with other measurable covariates (like trading volume), you should certainly try to surface those features and use them when flattening the sampling. $\endgroup$
    – Bill Bradley
    Commented Nov 20, 2022 at 13:09
  • 1
    $\begingroup$ (And again, please feel free to reach out to me directly-- I suspect the details of your data might guide the analysis, and that's probably beyond the scope of a website like this.) $\endgroup$
    – Bill Bradley
    Commented Nov 20, 2022 at 13:10
  • $\begingroup$ That's very helpful, thanks Bill! PS: I don't know how to PM you... $\endgroup$
    – sakuragaoka2001
    Commented Nov 20, 2022 at 15:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .