Contents

• Introduction
  • Randomness
  • Hash functions
  • Hash tables
  • Cuckoo hashing
• Probabilistic data structures
  • Bloom filter
  • Linear counter
  • Loglog counter
  • Count-min sketch
• Without hashing
  • Stream summary
  • Skip lists
  • Rabin–Karp pattern matching
  • Miller–Rabin primality check
  • Randomized min-cut algorithm
  • Randomized quicksort
• Machine Learning
  • Random tree classifier
  • Random forest
  • Low-density parity check codes
  • Digital fountains

Key themes of this talk

There's no perfect machine. Randomness serves different purposes. Accuracy can be traded for other resources.
(for example memory or speed)

Randomness

• a probability distribution (uniform = highest entropy)
• a degree of predictability (from deterministic to non-deterministic)

Unpredictability

• nature; thermal noise, shot noise, clock drift, particle emissions, etc
• humans; e.g. keystroke latencies of typed text, arrival times of ethernet packets

Chaotic behaviour

• hash functions: maps input to chaos
• pseudo-random number generators^*: fixed amount of memory, initial conditions = the seed.
• lossless compression algorithms: shorter description ⇒ increased entropy ⇒ chaos rslts al2 3mF0=^.
• binary expansions of certain numbers, e.g. π.

Uses of randomness

• Use unpredictability to thwart adversaries, e.g. to guard against worst-case attacks.
  Sources: nature, one-time pads.
  Used in: randomized quicksort, skip lists, seeds for PRNGs, generation of cryptographic keys, uses in security.
• Use chaotic behaviour for balance and fairness guarantees.
  Sources: Pseudo-random generators, hash functions, \(\pi\)
  Used in: MCMC. Generating random game worlds (that we'll want to re-generate again next time).

Hash functions

• Convert any input to a fixed-length, deterministic, chaotic output.
  Small change in input ⇒ drastically different hash.
• Hash functions are one-way functions: computationally impractical to reverse.
• Cryptographic hash functions make it infeasible to find any input matching a given hash.

Hash functions: there are lots

• Non-cryptographic: FNV hash, Murmur hash, CityHash, etc.
• Cryptographic: MD5, SHA–1, SHA–256, SHA–512, Keccak, etc.

Hash functions: example

Cryptographic hash functions: Quiz

Hash maps / hash tables

• a hash function maps items to array positions
• store something at that position (e.g. a linked list of associated values)

Arrays and hashes

Linear Counter

• Initialise a bit array \(\list{D}\) with zeros.
• Hash each \(x \in \set{X}\) to a location in \(\list{D}\), set those bits to 1.
• cardinality\((\set{X}) \approx \sum_i \list{D}[i] \)

Linear Counter: Example

Loglog Counter

• Map each \(x \in \set{X}\) to a hash.
  (The hash is a random bit string.)
• Out of \(\setsize{\set{X}}\) random bit strings, there are \(\approx \frac{\setsize{\set{X}}}{2^k}\) with \(k\) leading zeros
• Find \(K\), the largest number of leading zeros found in any of the \(\setsize{\set{X}}\) hashes.
• Use \(K\) to compute a very noisy estimate of \(\setsize{\set{X}}\).

Bloom filter

Bloom filter: Summary

• Approximate summary of set membership.
• Operation:
  
  • Set membership \(\in \set{X}\) is summarised in a bit array of length \(\hl{d}\).
  • Choose \(\hl{k}\) hash functions   (mapping elements \(\in\set{X}\) to integers \(\bset{0 \dots d}\))
    The \(\hl{k}\) hashes associate each \(x \in \set{X}\) with \(\hl{k}\) locations in the bit array.
  • To add an element \(x\): set the \(\hl{k}\) bits associated with \(x\) to 1.
  • To check \(x\) is in the filter: check if the associated \(\hl{k}\) bits are all 1.
• Features:
  Fast query speed: \(\bigO{k}\) Low storage: \(\bigO{d}\) No false-negatives Collisions lead to false-positives with
  \( \displaystyle \p{ x\in\set{B} \| x\notin\set{X}} =\left(1-\left(1-\frac{1}{d}\right)^{k\setsize{\set{X}}/d}\right)^{k} \ \approx\ \left(1-\exp\left(-\frac{k\setsize{\set{X}}}{d}\right)\right)^{k} \)

Count-min sketch

• Applications: Estimate the number of appearances of an item, Data range estimation
• Operation:
  • Original set has a large number of elements \(\setsize{\set{X}}\)
  • Choose \(\hl{K}\) hash functions that map to \( \hl{D} \) different positions
  • Use a \( \hl{K} \times \hl{D} \) matrix \(\matrix{A}\) to estimate occurences of items in the original set
  • Where \(\matrix{A}_{\hl{k},\hl{d}}\) is the number of times hash function \(\hl{k}\) mapped to position \( \hl{d} \)
• Features:
  • Low storage: \(\bigO{\hl{K}\hl{D}}\lll\bigO{\setsize{\set{X}}}\)
  • Estimation error: \( P(\text{error} \leq \frac{2\setsize{\set{X}}}{\hl{K}}) = 1 - \left(\frac{1}{2}\right)^\hl{D} \)
  • Term frequency estimate: \( \text{freq}(x) \approx \min \{h_i(x)\}^{\hl{K}}_{i = 1} \)

Cuckoo hashing

Monte Carlo tree search game playing

Monte Carlo tree search game playing

Monte Carlo tree search game playing

Monte Carlo tree search game playing

Monte Carlo tree search: Summary

• Applications:
  • Arbitrary and combinatorial game playing
  • Predictive control
  • Asymptotically complete global optimisation
• Operation:
  • Can be described as a 4-step algorithm
  • Selection: Choose optimal path in tree recursively until leaf node is reached
  • Expansion: Add one or more nodes according to possible actions
  • Simulation: Run simulation based on a default policy to produce an outcome
  • Backpropagation: Simulation results are used to update path statistics
• Features:
  • Straight-forward implementation
  • Many variants exist

Digital Fountain codes: Encoding

Digital Fountain codes: Decoding

Digital Fountains: Summary

• Applications:
  • Fast transmission of large amounts of data with little error correction.
  • Streaming and cloud services
• Features:
  Recover \(K\) symbols from \(N\) transmitted

  Algorithm	Encoding/decoding
  LT codes	\( \bigO{ K \mul \ln(\frac{K}{\delta}) } \)
  Solomon-Reed codes	\( \bigO{ K(N-K) \mul \log_2 N } \)

  Typical redundancy is roughly 5% in practice Equivalent encode/decode with a random sparse matrix

Digital Fountains: Operation

• Encode a bit array \(\list{S}\) into \(\list{T}\):
  • For each \(\hl{s}\)
    • Sample the degree \(\hl{d}\) of the encoding symbol from a degree distribution
    • Choose uniformly at random \(\hl{d}\) distinct elements of \(\list{S}\) as neighbors to \(\hl{s}\)
    • Set \(\hl{t}\) to the bitwise sum of \(\hl{s}\) neighbors
• Decode the received bit array \(\list{T}\):
  • For each \(\hl{s} \in \list{S}\)
    • Find a \(\hl{s}\) with only one neighbor and recover its value directly
    • Bitwise add the recovered \(\hl{s}\) to all \(\hl{t}\) that have \(\hl{s}\) as its neighbor
    • Remove \(\hl{s}\) from the list of neighbors of all other elements of \(\list{S}\)

Random tree classifier

Random forests

• Applications: classification, clustering, regression, data mining, ...
• Operation:
  • A random forest is a collection of decision trees.
  • Each decision tree in the forest is trained on a randomly sampled collection of data points.
    • Form \(\set{X}\) by sampling \(\setsize{\set{D}}\) elements from \(\set{D}\).
    • Generate a decision tree for \(\set{X}\).
  • The forest's prediction is the average of the individual tree predictions.
• Features:
• Ensemble method

Rabin–Karp pattern matching

• Approximate pattern search by comparing hashes rather than strings (using a rolling hash function for efficiency).
• Operation:
  • Compute the hash of the pattern \(\list{s}\) that is searched for.
  • Search the main text for hash matches:
    • Maintain a rolling hash of the last \(\len{\list{s}}\) characters, and check if it matches the hash of the pattern.
    • If they match, verify with string comparison.
  • efficient sequential hashing of the substrings in the main text is possible with a rolling hash function.
• Features:
  • Generalises to e.g. 2-dimensional pattern search
  • Faster than e.g. Knuth–Morris–Pratt, Boyer–Moore, especially for long patterns

Miller–Rabin primality test: Summary

• Applications: cryptography, information security
• Operation:
  • To check if \(\hl{x}\) is prime:
    
    • Randomly sample \(\hl{N}\) elements from the set \(\{1,\ldots, \hl{x} - 1 \}\)
  • For each sampled element \(y_n\), check if it is composite by:
    • Checking if \(y_n\) is Fermat witness
    • Checking if \(y_n\) satisfies \(y_n^2\equiv 1 \pmod{\hl{x}}\), and \(y_n\not\equiv \pm 1 \pmod{\hl{x}} \)
• Features:

  Algorithm	Time
  Randomized Miller–Rabin	\(\bigO{\log^3 \hl{x}}\)
  Deterministic Miller–Rabin	\(\bigO{\log^5 \hl{x}}\)
  Adleman–Pomerance–Rumely	\(\bigO{(\log \hl{x})^{\bigO{\log\log\log \hl{x}}}}\)

  • No false negatives; false positives decrease with \((\frac{1}{4})^\hl{N}\)
  • Does not rely on the Extended Riemann's hypothesis.

Low-density parity-check codes

• Applications: communication over noisy channels
• Operation:
  • Using a parity-check matrix \(H\) with \(m\) non-zero elements per row
  • Do a Bayesian reconstruction of bit array \(t\) from noisy \(r\) with:
    • Prior: \(P(t) \propto \one{Ht=0 \pmod 2} \)
    • Likelihood: \(P(r|t) \propto \prod_m \one{\sum_n H_{m,n}t_{n}=0 \pmod 2} \)
  • By applying belief propagation
• Features:
  • Near Shannon limit
  • Negligible undetectable errors in practice

Min-cut

Randomized Min-Cut

• Applications:
  • Minimal cardinality cut set in a graph; clustering, network design and reliability
• Operation:
  • In a graph with \(n\) nodes, repeat until only 2 nodes are left:
    • Choose an edge uniformly at random
    • Merge edge nodes

Lower bound on \(P(\text{final set is minimal})\)
Single run	\( \frac{2}{n(n-1)} \)
\(k\) runs	\(1 - \left( 1 -\frac{2}{n(n-1)} \right)^k\)

• Features:
  • Lower bounded success grows with multiple runs

Randomized Quicksort

• Sorts an array \(\list{A}\) of elements.
• Operation:
  • If \(\len{\list{A}} = 1\), return.
  • Select an index \(p\) uniformly at random from \(1 \dots \len{\list{A}}\).
  • Re-arrange the array so that:
    • elements \(\lt \list{A}[p]\) are in positions \(1 \ \mdots\ p\?-1\)
    • elements \(\ge \list{A}[p]\) are in positions \(p\?+1 \ \mdots\ \len{\list{A}}\)
  • Recursively sort \(\list{A}[1 \mdots p\?-1]\) and \(\list{A}[p\?+1 \mdots \len{\list{A}}]\)
  • The array \(\list{A}\) is now sorted!
• Features:
  worst case runtime: \(\textstyle \bigO{N^2}\) expected runtime: \(\bigO{N \log N}\), and better constant than many other sorting algorithms Randomization safeguards against repeatedly choosing poor splitting pivots

Heaps

Randomized Search Trees and Heaps (Treaps)

Randomized Search Trees and Heaps (Treaps)

• Applications:
• Sorting structure with balanced allocation
• Operation:
• For each new value \(x\)
  • Sample uniformly at random a priority \(y\)
  • Insert node \((x,y)\) keeping binary tree and heap properties
• Features:
• Low balancing cost

Skip lists

Skip lists: Summary

• Applications: Sorting
• Operation:
• Structure is composed of \(N\) ordered linked lists \(\set{X}\)
• To insert element \(y\)
  • Choose uniformly at randoma list level \(n\)
  • Find the largest \(x \in \set{X_n}\) for which \(x < y\) and insert \(y\) after \(x\)
  • Repeat the last step for all levels greater than \(n\)
• Searching:
  • For all \(n\)
    • Find the largest \(x \in \set{X_n}\) for which \(x \leq y\)
    • If \(y \neq x\), \(n \assign n + 1 \)
• Features:
  • Easier implementation than balanced trees
  • Balancing is independent of tree input sequence
  • Search cost: \(\bigO{\log n}\)

Stream Summary

Stream Summary

Stream Summary

• Estimate frequent items and top-\(k\) items in a data stream.
• Operation:
  • Maintain a fixed-length list \(\list{X}\) of tuples \((x,c,\varepsilon)\), where
    • \(x\) are labels
    • \(c\) are estimated number of occurences
    • \(\varepsilon\) is the maximal overestimation error
  • If the next label in the stream, \(\hl{x_{new}}\), is not in \(\list{X}\):
    • Find the tuple with least occurences \(c_{min}\)
    • Replace this tuple with \((\hl{x_{new}}, c_{min}+1, c_{min})\)
• Features:
  • Pre-defined, small storage
  • No underestimation — only overestimation
  • Maximum error is tracked online

Recap: use of randomness

Arrays with hash-mapped positions

Recommended reading

Probability and Computing: Randomized Algorithms and Probabilistic Analysis. Mitzenmacher & Upfal, 2005.

Entry-level textbook (non-free, few references).

Probabilistic Data Structures for Web Analytics and Data Mining. Blog post. Ilya Katsov (2012)

Nice motivation and presentation of various probabilistic data structures, including Bloom filter, count-min sketch, loglog counter, stream summary and others.