Elasticsearch Basics

An inverted index maps each term to the list of documents containing it (plus positions and frequencies), so a search looks up terms directly instead of scanning every document. Each Elasticsearch shard is a Lucene index made of immutable segments, each with its own inverted index.

• Indexing: text is analyzed into terms, then added to the term dictionary and postings lists.
• Query: look up each query term, intersect or union postings lists, then score.
• Segments are immutable: updates write a new document version and mark the old one deleted; background merges purge deleted docs.
• Doc values (columnar, on disk) serve sorting and aggregations, because the inverted index is poor at "value of field for doc".

An analyzer = character filters, one tokenizer, then token filters (lowercase, stop words, stemming). The same analysis must produce comparable terms at index and search time; mismatches or a keyword field cause misses.

POST /_analyze
{ "analyzer": "english", "text": "Running Shoes" }

Expected tokens: run, shoe.

• standard analyzer (default): splits on word boundaries, lowercases, no stemming: "Running" gives running.
• A keyword field is not analyzed, so a match for "nike air" will not hit a stored Nike-Air unless the value is queried exactly with term.
• Different analyzer and search_analyzer are used for edge n-gram autocomplete (index n-grams, search whole term).
• Changing an analyzer on an existing field needs a reindex; terms already in segments do not change.

Dynamic mapping guesses types from the first value seen (a string becomes text with a keyword sub-field, a date-like string becomes date), which can lock in a wrong type or create thousands of fields. Define explicit mappings with dynamic: strict for important indices.

PUT /products
{ "mappings": { "dynamic": "strict",
  "properties": {
    "name":  { "type": "text", "fields": { "raw": { "type": "keyword" } } },
    "sku":   { "type": "keyword" },
    "price": { "type": "scaled_float", "scaling_factor": 100 } } } }

• A field's type cannot be changed once data exists; you must create a new index and _reindex.
• Mapping explosion: user-supplied keys as field names hit index.mapping.total_fields.limit (default 1000); use flattened or a key/value nested structure.
• Use keyword for IDs, enums, and exact filters; text for full-text; scaled_float/long for money, not float.

BM25 (default since 5.0) scores a document higher for terms with high frequency in that document (saturating), rarer terms in the corpus (IDF), and shorter fields (length normalization), as opposed to older TF-IDF which grew unboundedly with term frequency.

• Tune with boost per field (multi_match with "fields": ["name^3","description"]), function_score for popularity or recency, and parameters k1 (saturation) and b (length normalization) on the similarity.
• Use filter context for criteria that should not influence score; filters are also cached.
• Inspect with "explain": true or the _explain API.
• IDF is computed per shard by default, so with small or skewed data scores differ between shards (search_type=dfs_query_then_fetch fixes this at a cost).

Primary shard count is fixed at index creation (changing it needs split/shrink or reindex); replicas can change anytime. Too many small shards waste heap and slow searches; too few limit parallelism and growth.

• Rough guidance: shards of about 10-50 GB, and keep shard count per node modest (Elastic suggests no more than ~20 shards per GB of heap).
• Each replica is a full copy: it gives failover and extra read throughput, but increases storage and indexing cost.
• A search fans out to every shard of the index and merges results, so oversharding multiplies overhead per query.
• For time-series, use data streams / ILM rollover so each backing index has a bounded size.
• Documents route to shards by hash(_routing) % number_of_primary_shards, which is why the count can't simply change.

Elasticsearch is near-real-time: new documents are in an in-memory buffer and become searchable only after a refresh, which by default happens every 1 second on indices that were searched recently. GET by ID is real-time, but search is not.

• PUT /idx/_doc/1?refresh=wait_for blocks until the next refresh makes it visible; refresh=true forces an immediate refresh (expensive if overused).
• Durability is separate: the translog is fsynced (per request by default) and flush commits segments to disk.
• For bulk loads, set index.refresh_interval to -1 (or 30s) and re-enable afterwards for much higher throughput.

curl -XPUT 'localhost:9200/orders/_doc/1?refresh=wait_for' -H 'Content-Type: application/json' -d '{"status":"NEW"}'

Use must for scoring text clauses, filter for exact conditions, should for optional boosts and must_not for exclusions.

GET /products/_search
{ "query": { "bool": {
    "must":   [ { "match": { "name": "pixel" } } ],
    "filter": [ { "term":  { "category": "phones" } },
                { "range": { "price": { "lt": 500 } } },
                { "term":  { "in_stock": true } } ],
    "should": [ { "range": { "released": { "gte": "now-1y" } } } ]
} } }

• With a must or filter present, should clauses only add score (minimum_should_match defaults to 0); with only should, at least one must match.
• filter and must_not run in filter context: no scoring, cacheable.
• match analyzes the text; term does not, so use term only on keyword/numeric/boolean fields.

from + size makes each shard return its top from + size hits to the coordinator, which sorts them, so cost grows with depth; beyond index.max_result_window (10,000) it is rejected. Use search_after with a stable sort and a point in time (PIT).

POST /_search
{ "size": 20,
  "pit": { "id": "<pit-id>", "keep_alive": "1m" },
  "sort": [ { "created": "desc" }, { "_shard_doc": "asc" } ],
  "search_after": [1759570000000, 4182] }

• Pass the sort values of the last hit as search_after; the PIT keeps a consistent snapshot while paging.
• The sort needs a unique tiebreaker (_shard_doc with PIT, or a unique id field), otherwise results can skip or repeat.
• Scroll API is for bulk export, no longer recommended for user pagination; search_after cannot jump to an arbitrary page.

Aggregations run over doc values; use a terms bucket with sub-aggregations. Set size: 0 to skip hits.

GET /products/_search
{ "size": 0, "aggs": {
  "by_cat": { "terms": { "field": "category", "size": 10 },
    "aggs": { "avg_price": { "avg": { "field": "price" } },
              "prices": { "histogram": { "field": "price", "interval": 100 } } } } } }

• terms is approximate over many shards: each shard returns its top candidates, so counts for lower buckets can be inexact (see doc_count_error_upper_bound); raise shard_size for accuracy.
• Aggregating on text fields is disallowed by default (needs fielddata, heap-heavy); use keyword.
• High-cardinality terms and cardinality (HyperLogLog++ approximation) cost memory; use composite aggregation to page through all buckets.

Treat the database as the source of truth and Elasticsearch as a derived, rebuildable view. Choose the sync mechanism by consistency needs.

• Dual write in the service: simplest but unreliable; if ES fails after the DB commit, they diverge.
• Transactional outbox: write an event row in the same DB transaction, a relay publishes to Kafka, a consumer indexes into ES. At-least-once delivery with idempotent indexing by document ID.
• CDC (Debezium) from the DB log into Kafka then ES sink: captures all writers, small lag.
• Periodic batch reindex by updated_at as a safety net.
• Use external versioning (version_type=external with the row version) so out-of-order events cannot overwrite newer data.
• Keep the ability to rebuild: write to a new index and switch an alias for zero-downtime reindex.

Split brain means two parts of a partitioned cluster each elect a master and accept writes, causing divergent data. Since Elasticsearch 7 the cluster coordination layer uses a quorum-based voting configuration managed automatically, so a minority partition cannot elect a master.

• Have 3 dedicated master-eligible nodes (odd number); the cluster stays available with one down.
• Pre-7 clusters needed discovery.zen.minimum_master_nodes = (N/2)+1 set manually; wrong values were the classic cause.
• Master-eligible nodes should be dedicated (not data nodes under heavy GC) so elections are stable.
• Minority-side nodes cannot accept cluster-state changes; searches on local shards may still serve stale data.

Green means all primaries and replicas are assigned; yellow means all primaries are assigned but some replicas are not (data is searchable, but no redundancy); red means at least one primary is unassigned, so data is missing for those shards.

GET _cluster/health
GET _cluster/allocation/explain
GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason

• A single-node cluster with replicas > 0 is permanently yellow, because a replica cannot share the primary's node.
• Common causes: disk watermarks exceeded (low 85%, high 90%, flood-stage 95% makes indices read-only), node left, allocation filters, shard limit per node.
• allocation/explain states exactly why a shard is unassigned.
• After disk flood-stage, free space then clear index.blocks.read_only_allow_delete (auto-cleared in recent versions once space is below high watermark).

Typical culprits are leading wildcards, large aggregations on high-cardinality fields, deep pagination, scripts, and fielddata on text fields.

• Leading wildcard *foo and regexp scan the whole term dictionary; use n-grams or the wildcard field type.
• terms aggregation or sort on text with fielddata loads all terms into heap; use keyword.
• Too many shards or segments, unmerged deletes; check _cat/segments, _nodes/stats.
• Use the slow log (index.search.slowlog.threshold.query.warn) and the profile API ("profile": true) to see per-clause timing.
• Large size or fetching huge _source; restrict with _source filtering and fields.

Do not use it as a transactional system of record or where you need strict consistency, joins or multi-row transactions. Use it for search, analytics over logs, and filtering large denormalized datasets.

• No ACID transactions across documents; refresh delay means reads after writes may miss.
• Joins are limited (nested, join fields are costly); model data denormalized, which pushes complexity to the sync layer.
• Simple LIKE search over small data is better served by database indexes or PostgreSQL full-text search/pg_trgm.
• Operating a cluster (shards, heap, upgrades, reindexing) has real cost; weigh it against managed services or DB-native search.
• Exact counts and aggregations over many shards can be approximate.

Plain object arrays are flattened, so field values from different items get mixed together and cross-matches appear; nested indexes each item as a hidden separate document, preserving per-item correlation at a cost.

{ "items": [ { "sku": "A", "qty": 1 }, { "sku": "B", "qty": 5 } ] }

A query for sku=A AND qty=5 on an object mapping wrongly matches this order; on nested with a nested query it does not.

• nested: query with nested query/aggregation; each item counts toward doc limits and indexing cost, and updating one item reindexes the whole root doc.
• join (parent-child): items updated independently, but queries are slower and need same-shard routing; use rarely.
• Often best is denormalization: duplicate the needed parent fields into each child document and search the child level.