User:Optimalsloth/sandbox

In computer science, a static function, also known as retrieval data structure, is a space-efficient dictionary-like data type composed of a collection of (key, value) pairs that allows the following operations:

• Construction from a collection of (key, value) pairs
• Retrieve the value associated with the given key or anything if the key is not contained in the collection
• Update the value associated with a key (optional)

They can also be thought of as a function $b\colon {\mathcal {U}}\to \{0,1\}^{r}$ for an universe ${\mathcal {U}}$ and the set of keys $S\subseteq {\mathcal {U}}$ where retrieve has to return $b(x)$ for any value $x\in S$ and a random value from $\{0,1\}^{r}$ otherwise.

In contrast to static functions, AMQ-filters support (probabilistic) membership queries and dictionaries additionally allow operations like listing keys or looking up the value associated with a key and returning some other symbol if the key is not contained.

As can be derived from the operations, this data structure does not need to store the keys at all and may actually use less space than would be needed for a simple list of the key value pairs. This makes it attractive in situations where the associated data is small (e.g. a few bits) compared to the keys because we can save a lot by reducing the space used by keys.

To give a simple example suppose $n$ names annotated with either male or female are given. A static function built from this database can reproduce the associated tag for all names contained in the original set and a random one for other names. The size of this static function can be made to be only $(1+\epsilon )n$ bits for a small $\epsilon$ which is obviously much less than any other pair based representation.

Examples

A trivial example of a static function is a sorted list of the keys and values which implements all the above operations and many more. However, the retrieve on a list is slow and we implement many unneeded operations that can be removed to allow optimizations. Furthermore, we are even allowed to return junk if the queried key is not contained which we did not use at all.

XOR-Filters

Hashed filters can be categorized by their queries into OR, AND and XOR-filters. For example, the bloom filter is an AND-filter since it returns true for a membership query if all probed locations match. XOR filters work only for static filters and are the most promising for building them space efficiently. They are built by solving a linear system which ensures that a query for every key returns true.

Construction

Given a hash function $h$ that maps each each key to a bitvector of length $m\geq \left\vert S\right\vert =n$ where all $(h(x))_{x\in S}$are linearly independant the following system of linear equations has a solution $Z\in \{0,1\}^{m\times r}$:

$(h(x)\cdot Z=b(x))_{x\in S}$

Therefore, the static function is given by $h$ and $Z$ and the space usage is roughly $(1+\epsilon )n$ bits per key for $m=(1+\epsilon )n$. A retrieval for $x\in {\mathcal {U}}$ can be expressed as the bitwise XOR of the rows $Z_{i}$ for all set bits $i$ of $h(x)$. Furthermore, fast queries require sparse $h(x)$, thus the problems that need to be solved for this method are finding a suitable hash function and still being able to solve the system of linear equations efficiently.

Ribbon fiters

(siehe "Ribbon filter: practically smaller than Bloom and Xor", Abschnitt 2). Kann auf verschiedene Arten konstruiert werden, zum Beispiel indem man lineare Gleichungssysteme löst. Das Paper beschreibt eine effiziente Möglichkeit, diese Gleichungssysteme aufzubauen und zu lösen, aber da würde ich vielleicht nur kurz die Idee (Werte im LGS hauptsächlich auf der Diagonalen) erwähnen und nicht das komplette Paper lesen/erklären. Ich persönlich finde das XOR-basierte retrieval den interessantesten Abschnitt.

Perfect Hash Functions

(Verlinken zum entsprechenden Wikipedia-Artikel). Man baut die PHF auf und speichert dann in einem Array an der Position vom Hash-Funktionswert die Daten. Mit einer Minimum Perfect Hash Function hat man dann (besonders bei großen abzurufenden Elementen) kaum Overhead. Es ist hier sogar möglich, die Values dynamisch zu ändern, aber die möglichen Keys sind weiterhin statisch.

Applications

Approximate membership

(ähnlich bloom filter) - Benutze das Element als Key und einen kurzen Hash als Value. Für einen Query muss man die static function auswerten und vergleichen, ob der Hash der gleiche ist. Die Länge des Hash steuert die false-positive-rate.

Perfect Hash Functions

Das wird jetzt ein bisschen rekursiv, weil es bei den Beispielen zur Konstruktion auch schon dran stand, aber man kann static functions dazu benutzen, PHF zu bauen. Stell dir zum Beispiel eine cuckoo hashtabelle vor, in die schon alle Elemente eingefügt sind. Jedes Element darf an 2 verschiedenen Positionen sitzen und an jeder Position darf nur ein Element sitzen. Um jetzt eine PHF dieser Elemente zu generieren, müssen wir uns pro Element nur noch speichern, welche seiner beiden Positionen es denn am Ende genommen hat. Also eine 1-bit static function.