Skip to main content
invertible bloom filter pointer
Source Link
David Eppstein
  • 18.6k
  • 2
  • 55
  • 127

It's at least possible to test whether the input is a permutation with a randomized algorithm that uses O(1) space, always answers "yes" when it is a permutation, and answers "yes" incorrectly when it is not a permutation only with very small probability.

Simply pick a hash function $h(x)$, compute $\sum_{i=1}^n h(i)$, compute $\sum_{i=1}^n h(a[i])$, and compare the two sums.

Ok, some care needs to be used in defining and choosing among an appropriate family of hash functions if you want a rigorous solution (and I suppose we do want one, since we're on mathoverflow not stackoverflow). Probably the simplest way is just to fill another array $H$ with random numbers and let $h(x)=H[x]$, but that is unacceptable because it uses too much space. I'll leave this part as unsolved and state this as a partial answer rather than claiming full rigor at this point.

See also my paper Space-Efficient Straggler Identification in Round-Trip Data Streams via Newton's Identitities and Invertible Bloom Filters which solves a more general problem (if there are O(1) duplicates, say which ones are duplicated, using only O(1) space) with the same lacuna in how the hash functions are defined. It also contains a proof that an algorithm that makes only a single pass over the data cannot solve the problem exactly and deterministically, but of course that doesn't apply to algorithms with random access to the input array.

It's at least possible to test whether the input is a permutation with a randomized algorithm that uses O(1) space, always answers "yes" when it is a permutation, and answers "yes" incorrectly when it is not a permutation only with very small probability.

Simply pick a hash function $h(x)$, compute $\sum_{i=1}^n h(i)$, compute $\sum_{i=1}^n h(a[i])$, and compare the two sums.

Ok, some care needs to be used in defining and choosing among an appropriate family of hash functions if you want a rigorous solution (and I suppose we do want one, since we're on mathoverflow not stackoverflow). Probably the simplest way is just to fill another array $H$ with random numbers and let $h(x)=H[x]$, but that is unacceptable because it uses too much space. I'll leave this part as unsolved and state this as a partial answer rather than claiming full rigor at this point.

It's at least possible to test whether the input is a permutation with a randomized algorithm that uses O(1) space, always answers "yes" when it is a permutation, and answers "yes" incorrectly when it is not a permutation only with very small probability.

Simply pick a hash function $h(x)$, compute $\sum_{i=1}^n h(i)$, compute $\sum_{i=1}^n h(a[i])$, and compare the two sums.

Ok, some care needs to be used in defining and choosing among an appropriate family of hash functions if you want a rigorous solution (and I suppose we do want one, since we're on mathoverflow not stackoverflow). Probably the simplest way is just to fill another array $H$ with random numbers and let $h(x)=H[x]$, but that is unacceptable because it uses too much space. I'll leave this part as unsolved and state this as a partial answer rather than claiming full rigor at this point.

See also my paper Space-Efficient Straggler Identification in Round-Trip Data Streams via Newton's Identitities and Invertible Bloom Filters which solves a more general problem (if there are O(1) duplicates, say which ones are duplicated, using only O(1) space) with the same lacuna in how the hash functions are defined. It also contains a proof that an algorithm that makes only a single pass over the data cannot solve the problem exactly and deterministically, but of course that doesn't apply to algorithms with random access to the input array.

Post Undeleted by David Eppstein
+$; added 61 characters in body; added 172 characters in body
Source Link
David Eppstein
  • 18.6k
  • 2
  • 55
  • 127

There are several easyIt's at least possible to test whether the input is a permutation with a randomized algorithm that uses O(n1) time solutions. Firstspace, since your numbers are all in the range 1..nalways answers "yes" when it is a permutation, you can bucket sort them in linear timeand answers "yes" incorrectly when it is not a permutation only with very small probability. Second, make an array of n binary values

Simply pick a hash function $h(x)$, all initially falsecompute $\sum_{i=1}^n h(i)$, scan your input setting the corresponding array values to truecompute $\sum_{i=1}^n h(a[i])$, and then check that everything has been set to true. Third, even if you wanted to detect duplicates in more complex input such as a set of real numbers, you could use a hash tablecompare the two sums.

TheoreticallyOk, there are some proofs thatcare needs to be used in limited modelsdefining and choosing among an appropriate family of computationhash functions if you want a rigorous solution (disallowing hash tables) with real number inputs one can'tand I suppose we do better than $\Omega(n\log n)$ for duplicate detection: see e.g. Yao, A. C.-C. (1989), "Lower bounds for algebraic computation trees with integer inputs", Proc. 30th Annual Symposium on Foundations of Computer Science (FOCS 1989), pp. 308–313want one, since we're on mathoverflow not stackoverflow). But these lower bounds don't apply to your problem becauseProbably the inputsimplest way is all integersjust to fill another array $H$ with random numbers and let $h(x)=H[x]$, but that is unacceptable because it uses too much space. I'll leave this part as unsolved and state this as a partial answer rather than claiming full rigor at this point.

There are several easy O(n) time solutions. First, since your numbers are all in the range 1..n, you can bucket sort them in linear time. Second, make an array of n binary values, all initially false, scan your input setting the corresponding array values to true, and then check that everything has been set to true. Third, even if you wanted to detect duplicates in more complex input such as a set of real numbers, you could use a hash table.

Theoretically, there are some proofs that in limited models of computation (disallowing hash tables) with real number inputs one can't do better than $\Omega(n\log n)$ for duplicate detection: see e.g. Yao, A. C.-C. (1989), "Lower bounds for algebraic computation trees with integer inputs", Proc. 30th Annual Symposium on Foundations of Computer Science (FOCS 1989), pp. 308–313. But these lower bounds don't apply to your problem because the input is all integers.

It's at least possible to test whether the input is a permutation with a randomized algorithm that uses O(1) space, always answers "yes" when it is a permutation, and answers "yes" incorrectly when it is not a permutation only with very small probability.

Simply pick a hash function $h(x)$, compute $\sum_{i=1}^n h(i)$, compute $\sum_{i=1}^n h(a[i])$, and compare the two sums.

Ok, some care needs to be used in defining and choosing among an appropriate family of hash functions if you want a rigorous solution (and I suppose we do want one, since we're on mathoverflow not stackoverflow). Probably the simplest way is just to fill another array $H$ with random numbers and let $h(x)=H[x]$, but that is unacceptable because it uses too much space. I'll leave this part as unsolved and state this as a partial answer rather than claiming full rigor at this point.

Post Deleted by David Eppstein
Source Link
David Eppstein
  • 18.6k
  • 2
  • 55
  • 127

There are several easy O(n) time solutions. First, since your numbers are all in the range 1..n, you can bucket sort them in linear time. Second, make an array of n binary values, all initially false, scan your input setting the corresponding array values to true, and then check that everything has been set to true. Third, even if you wanted to detect duplicates in more complex input such as a set of real numbers, you could use a hash table.

Theoretically, there are some proofs that in limited models of computation (disallowing hash tables) with real number inputs one can't do better than $\Omega(n\log n)$ for duplicate detection: see e.g. Yao, A. C.-C. (1989), "Lower bounds for algebraic computation trees with integer inputs", Proc. 30th Annual Symposium on Foundations of Computer Science (FOCS 1989), pp. 308–313. But these lower bounds don't apply to your problem because the input is all integers.