Read on Twitter
how does log file rotation map or compare to elasticsearch index lifecycle management (part of any recent @elastic release)? thread to clear up some confusion I& #39;ve seen recently 
https://abs.twimg.com/emoji/v2/... draggable="false" alt="🧵" title="Thread" aria-label="Emoji: Thread">
log file rotation with the example of elasticsearch itself ( https://github.com/elastic/elasticsearch/blob/master/distribution/src/config/log4j2.properties):">https://github.com/elastic/e...
* one or multiple conditions (daily + max 128MB files) when you roll over to the next file
* strategy when to delete the oldest file (max 2GB in sum)
* one or multiple conditions (daily + max 128MB files) when you roll over to the next file
* strategy when to delete the oldest file (max 2GB in sum)
elasticsearch and the kibana UI allow you to specify index lifecycle policies that manage linked indices like logs, metrics, traces, or any other time-based data
you start with the *hot phase* and can optionally add warm and cold after that plus a delete phase if you don& #39;t want to keep your data forever
the hot phase has a lot of similarities to file rotation conditions but is called rollover (to a new index) with any combination of size, # of documents, or age
the default condition is 30d + 50GB (single shard in that index) which is a good starting point for most cases
the default condition is 30d + 50GB (single shard in that index) which is a good starting point for most cases
optionally you can do some optimizations after the rollover: merge (lucene) segments, shrink the number of primary shards (in case you& #39;ve changed the default for better write parallelization), or make the index read-only. all of those you could do in the *warm phase* too
which leads to the multi-tier part that doesn& #39;t exist in file rotation
large amounts of data & long retention make different hardware profiles appealing: keep recent data on fast nodes where all writes & most reads happen. 1w after the rollover move to slower nodes with more disk
large amounts of data & long retention make different hardware profiles appealing: keep recent data on fast nodes where all writes & most reads happen. 1w after the rollover move to slower nodes with more disk
where 1w is an example — it could be after 24h or a month depending on your requirements
why can the optimizations (force merge, shrink) be either on hot or warm? you might have spare capacity on your faster hot nodes while others want to offload that extra work — "it depends"
why can the optimizations (force merge, shrink) be either on hot or warm? you might have spare capacity on your faster hot nodes while others want to offload that extra work — "it depends"
in the *cold phase* you can add yet another hardware profile (maybe spinning disks), freeze an index to minimize the resource footprint, or even offload the data to a blob store like S3 with a searchable snapshot (the only paid feature in this thread)
in the *delete phase* you can remove data (365d after the rollover here). optionally after taking a final snapshot which is controlled by — you& #39;ve guessed it — a snapshot lifecycle policy
it& #39;s kind of similar to log file rotation but can only be based on time, not storage size
it& #39;s kind of similar to log file rotation but can only be based on time, not storage size
while no disk space based deletion might be a limitation to some, it makes your retention more predictable (also for audits and compliance) and a spike in logs cannot wipe out a month of past data
also there is now disk-based autoscaling, but that& #39;s for another thread...
also there is now disk-based autoscaling, but that& #39;s for another thread...
