Skip to content

Releases: prometheus/prometheus

3.5.0-rc.1 / 2025-07-08

11 Jul 10:19
v3.5.0-rc.1
eea2037
Compare
Choose a tag to compare
Pre-release

3.5 will be a Long Term Support (LTS) release; see release https://prometheus.io/docs/introduction/release-cycle/
There are a number of new features, and no known breaking changes in this release.

The following feature was removed since RC.1, since problems were discovered:

  • [FEATURE] OTLP: Support promoting OTel scope name/version/schema URL/attributes as metric labels, via otlp.promote_scope_metadata. #16730 #16760

What's Changed

  • [FEATURE] PromQL: Add experimental type and unit metadata labels, behind feature flag type-and-unit-labels. #16228 #16632 #16718 #16743
  • [FEATURE] PromQL: Add ts_of_(min|max|last)_over_time, behind feature flag experimental-promql-functions. #16722 #16733
  • [FEATURE] Scraping: Add global option always_scrape_classic_histograms to scrape a classic histogram even if it is also exposed as native. #16452
  • [FEATURE] OTLP: New config options promote_all_resource_attributes and ignore_resource_attributes. #16426
  • [FEATURE] Discovery: New service discovery for STACKIT Cloud. #16401
  • [ENHANCEMENT] Hetzner SD: Add label_selector to filter servers. #16512
  • [ENHANCEMENT] PromQL: support non-constant parameter in aggregations like quantile and topk. #16404
  • [ENHANCEMENT] UI: Better total target count display when using keep_dropped_targets option. #16604
  • [ENHANCEMENT] UI: Add simple filtering on the /rules page. #16605
  • [ENHANCEMENT] UI: Display query stats in hover tooltip over table query tab. #16723
  • [ENHANCEMENT] UI: Clear search field on /targets page. #16567
  • [ENHANCEMENT] Rules: Check that rules parse without error earlier at startup. #16601
  • [ENHANCEMENT] Promtool: Optional fuzzy float64 comparison in rules unittests. #16395
  • [PERF] PromQL: Reuse histogramStatsIterator where possible. #16686
  • [PERF] PromQL: Reuse storage for custom bucket values for native histograms. #16565
  • [PERF] UI: Optimize memoization and search debouncing on /targets page. #16589
  • [PERF] UI: Fix full-page re-rendering when opening status nav menu. #16590
  • [PERF] Kubernetes SD: use service cache.Indexer to achieve better performance. #16365
  • [PERF] TSDB: Optionally use Direct IO for chunks writing. #15365
  • [PERF] TSDB: When fetching label values, stop work earlier if the limit is reached. #16158
  • [PERF] Labels: Simpler/faster stringlabels encoding. #16069
  • [PERF] Scraping: Reload scrape pools concurrently. #16595 #16783
  • [BUGFIX] Top-level: Update GOGC before loading TSDB. #16491
  • [BUGFIX] Config: Respect GOGC environment variable if no "runtime" block exists. #16558
  • [BUGFIX] PromQL: Fix native histogram last_over_time. #16744
  • [BUGFIX] PromQL: Fix reported parser position range in errors for aggregations wrapped in ParenExpr #16041 #16754
  • [BUGFIX] PromQL: Don't emit a value from histogram_fraction or histogram_quantile if classic and native histograms are present at the same timestamp. #16552
  • [BUGFIX] PromQL: Incorrect rounding of [1001ms] to [1s] and similar. #16478
  • [BUGFIX] PromQL: Fix inconsistent / sometimes negative histogram_count and histogram_sum. #16682
  • [BUGFIX] PromQL: Improve handling of NaNs in native histograms. #16724
  • [BUGFIX] PromQL: Fix unary operator precedence in duration expressions. #16713
  • [BUGFIX] PromQL: Improve consistency of avg aggregation and avg_over_time. #16569 #16773
  • [BUGFIX] UI: Add query warnings and info to graph view. #16753 #16759
  • [BUGFIX] API: Add HTTP Vary: Origin header to responses to avoid cache poisoning. #16008
  • [BUGFIX] Discovery: Avoid deadlocks by taking locks in consistent order. #16587
  • [BUGFIX] Remote-write: For Azure AD auth, allow empty client_id to suppport system assigned managed identity. #16421
  • [BUGFIX] Scraping: Fix rare memory corruption bug. #16623
  • [BUGFIX] Scraping: continue handling custom-bucket histograms after an exponential histogram is encountered. #16720
  • [BUGFIX] OTLP: Default config not respected when otlp: block is unset. #16693

3.5.0-rc.0 / 2025-06-25

30 Jun 15:04
v3.5.0-rc.0
31f0c70
Compare
Choose a tag to compare
Pre-release

3.5 will be a Long Term Support (LTS) release; see release https://prometheus.io/docs/introduction/release-cycle/
There are a number of new features, and no known breaking changes in this release.

What's Changed

  • [FEATURE] PromQL: Add experimental type and unit metadata labels, behind feature flag type-and-unit-labels. #16228 #16632 #16718 #16743
  • [FEATURE] PromQL: Add ts_of_(min|max|last)_over_time, behind feature flag experimental-promql-functions. #16722 #16733
  • [FEATURE] Scraping: Add global option always_scrape_classic_histograms to scrape a classic histogram even if it is also exposed as native. #16452
  • [FEATURE] OTLP: Support promoting OTel scope name/version/schema URL/attributes as metric labels, via otlp.promote_scope_metadata. #16730 #16760
  • [FEATURE] OTLP: New config options promote_all_resource_attributes and ignore_resource_attributes. #16426
  • [FEATURE] Discovery: New service discovery for STACKIT Cloud. #16401
  • [ENHANCEMENT] Hetzner SD: Add label_selector to filter servers. #16512
  • [ENHANCEMENT] PromQL: support non-constant parameter in aggregations like quantile and topk. #16404
  • [ENHANCEMENT] UI: Better total target count display when using keep_dropped_targets option. #16604
  • [ENHANCEMENT] UI: Add simple filtering on the /rules page. #16605
  • [ENHANCEMENT] UI: Display query stats in hover tooltip over table query tab. #16723
  • [ENHANCEMENT] UI: Clear search field on /targets page. #16567
  • [ENHANCEMENT] Rules: Check that rules parse without error earlier at startup. #16601
  • [ENHANCEMENT] Promtool: Optional fuzzy float64 comparison in rules unittests. #16395
  • [PERF] PromQL: Reuse histogramStatsIterator where possible. #16686
  • [PERF] PromQL: Reuse storage for custom bucket values for native histograms. #16565
  • [PERF] UI: Optimize memoization and search debouncing on /targets page. #16589
  • [PERF] UI: Fix full-page re-rendering when opening status nav menu. #16590
  • [PERF] Kubernetes SD: use service cache.Indexer to achieve better performance. #16365
  • [PERF] TSDB: Optionally use Direct IO for chunks writing. #15365
  • [PERF] TSDB: When fetching label values, stop work earlier if the limit is reached. #16158
  • [PERF] Labels: Simpler/faster stringlabels encoding. #16069
  • [PERF] Scraping: Reload scrape pools concurrently. #16595 #16783
  • [BUGFIX] Top-level: Update GOGC before loading TSDB. #16491
  • [BUGFIX] Config: Respect GOGC environment variable if no "runtime" block exists. #16558
  • [BUGFIX] PromQL: Fix native histogram last_over_time. #16744
  • [BUGFIX] PromQL: Fix reported parser position range in errors for aggregations wrapped in ParenExpr #16041 #16754
  • [BUGFIX] PromQL: Don't emit a value from histogram_fraction or histogram_quantile if classic and native histograms are present at the same timestamp. #16552
  • [BUGFIX] PromQL: Incorrect rounding of [1001ms] to [1s] and similar. #16478
  • [BUGFIX] PromQL: Fix inconsistent / sometimes negative histogram_count and histogram_sum. #16682
  • [BUGFIX] PromQL: Improve handling of NaNs in native histograms. #16724
  • [BUGFIX] PromQL: Fix unary operator precedence in duration expressions. #16713
  • [BUGFIX] PromQL: Improve consistency of avg aggregation and avg_over_time. #16569 #16773
  • [BUGFIX] UI: Add query warnings and info to graph view. #16753 #16759
  • [BUGFIX] API: Add HTTP Vary: Origin header to responses to avoid cache poisoning. #16008
  • [BUGFIX] Discovery: Avoid deadlocks by taking locks in consistent order. #16587
  • [BUGFIX] Remote-write: For Azure AD auth, allow empty client_id to suppport system assigned managed identity. #16421
  • [BUGFIX] Scraping: Fix rare memory corruption bug. #16623
  • [BUGFIX] Scraping: continue handling custom-bucket histograms after an exponential histogram is encountered. #16720
  • [BUGFIX] OTLP: Default config not respected when otlp: block is unset. #16693

2.53.5 / 2025-06-27

30 Jun 11:01
v2.53.5
d344ea7
Compare
Choose a tag to compare

[LTS patch release]

  • [ENHANCEMENT] TSDB: Add backward compatibility with the upcoming TSDB block index v3 #16762
  • [BUGFIX] Top-level: Update GOGC before loading TSDB #16521

3.4.2 / 2025-06-26

26 Jun 21:45
v3.4.2
b392caf
Compare
Choose a tag to compare
  • [BUGFIX] OTLP receiver: Fix default configuration not being respected if the otlp: block is unset in the config file. #16693

3.4.1 / 2025-05-31

31 May 13:45
v3.4.1
aea6503
Compare
Choose a tag to compare
  • [BUGFIX] Parser: Add reproducer for a dangling-reference issue in parsers. #16633

3.4.0 / 2025-05-17

17 May 07:28
v3.4.0
546b1d2
Compare
Choose a tag to compare
  • [CHANGE] Config: Make setting out-of-order native histograms feature (--enable-feature=ooo-native-histograms) a no-op. Out-of-order native histograms are now always enabled when out_of_order_time_window is greater than zero and --enable-feature=native-histograms is set. #16207
  • [FEATURE] OTLP translate: Add feature flag for optionally translating OTel explicit bucket histograms into native histograms with custom buckets. #15850
  • [FEATURE] OTLP translate: Add option to receive OTLP metrics without translating names or attributes. #16441
  • [FEATURE] PromQL: allow arithmetic operations in durations in PromQL parser. #16249
  • [FEATURE] OTLP receiver: Add primitive support for ingesting OTLP delta metrics as-is. #16360
  • [ENHANCEMENT] PromQL: histogram_fraction for bucket histograms. #16095
  • [ENHANCEMENT] TSDB: add prometheus_tsdb_wal_replay_unknown_refs_total and prometheus_tsdb_wbl_replay_unknown_refs_total metrics to track unknown series references during WAL/WBL replay. #16166
  • [ENHANCEMENT] Scraping: Add config option for escaping scheme request. #16066
  • [ENHANCEMENT] Config: Add global config option for convert_classic_histograms_to_nhcb. #16226
  • [ENHANCEMENT] Alerting: make batch size configurable (--alertmanager.notification-batch-size). #16254
  • [PERF] Kubernetes SD: make endpointSlice discovery more efficient. #16433
  • [BUGFIX] Config: Fix auto-reload on changes to rule and scrape config files. #16340
  • [BUGFIX] Scraping: Skip native histogram series if ingestion is disabled. #16218
  • [BUGFIX] TSDB: Handle metadata/tombstones/exemplars for duplicate series during WAL replay. #16231
  • [BUGFIX] TSDB: Avoid processing exemplars outside the valid time range during WAL replay. #16242
  • [BUGFIX] Promtool: Add feature flags for PromQL features. #16443
  • [BUGFIX] Rules: correct logging of alert name & template data. #15093
  • [BUGFIX] PromQL: Use arithmetic mean for histogram_stddev() and histogram_stdvar() . #16444

3.4.0-rc.0 / 2025-04-29

02 May 18:12
v3.4.0-rc.0
7a92230
Compare
Choose a tag to compare
Pre-release
  • [CHANGE] Config: Make setting out-of-order native histograms feature (--enable-feature=ooo-native-histograms) a no-op. Out-of-order native histograms are now always enabled when out_of_order_time_window is greater than zero and --enable-feature=native-histograms is set. #16207
  • [FEATURE] OTLP translate: Add feature flag for optionally translating OTel explicit bucket histograms into native histograms with custom buckets. #15850
  • [FEATURE] OTLP translate: Add option to receive OTLP metrics without translating names or attributes. #16441
  • [FEATURE] PromQL: allow arithmetic operations in durations in PromQL parser. #16249
  • [FEATURE] OTLP receiver: Add primitive support for ingesting OTLP delta metrics as-is. #16360
  • [ENHANCEMENT] PromQL: histogram_fraction for bucket histograms. #16095
  • [ENHANCEMENT] TSDB: add prometheus_tsdb_wal_replay_unknown_refs_total and prometheus_tsdb_wbl_replay_unknown_refs_total metrics to track unknown series references during WAL/WBL replay. #16166
  • [ENHANCEMENT] Scraping: Add config option for escaping scheme request. #16066
  • [ENHANCEMENT] Config: Add global config option for convert_classic_histograms_to_nhcb. #16226
  • [ENHANCEMENT] Alerting: make batch size configurable (--alertmanager.notification-batch-size). #16254
  • [PERF] Kubernetes SD: make endpointSlice discovery more efficient. #16433
  • [BUGFIX] Config: Fix auto-reload on changes to rule and scrape config files. #16340
  • [BUGFIX] Scraping: Skip native histogram series if ingestion is disabled. #16218
  • [BUGFIX] TSDB: Handle metadata/tombstones/exemplars for duplicate series during WAL replay. #16231
  • [BUGFIX] TSDB: Avoid processing exemplars outside the valid time range during WAL replay. #16242
  • [BUGFIX] Promtool: Add feature flags for PromQL features. #16443
  • [BUGFIX] Rules: correct logging of alert name & template data. #15093
  • [BUGFIX] PromQL: Use arithmetic mean for histogram_stddev() and histogram_stdvar() . #16444

3.3.1 / 2025-05-02

02 May 15:41
v3.3.1
3dcecab
Compare
Choose a tag to compare
  • [BUGFIX] Azure SD: Fix panic on malformed log message. #16434 #16210
  • [BUGFIX] Config: Update GOGC before loading TSDB. #16491

3.3.0 / 2025-04-15

18 Apr 13:46
v3.3.0
3c49e69
Compare
Choose a tag to compare
  • [FEATURE] PromQL: Implement idelta() and irate() for native histograms. #15853
  • [ENHANCEMENT] Scaleway SD: Add __meta_scaleway_instance_public_ipv4_addresses and __meta_scaleway_instance_public_ipv6_addresses labels. #14228
  • [ENHANCEMENT] TSDB: Reduce locking while reloading blocks. #12920
  • [ENHANCEMENT] PromQL: Allow UTF-8 labels in label_replace(). #15974
  • [ENHANCEMENT] Promtool: tsdb create-blocks-from openmetrics can now read from a Pipe. #16011
  • [ENHANCEMENT] Rules: Add support for anchors and aliases in rule files. #14957
  • [ENHANCEMENT] Dockerfile: Make /prometheus writable. #16073
  • [ENHANCEMENT] API: Include scrape pool name for dropped targets in /api/v1/targets. #16085
  • [ENHANCEMENT] UI: Improve time formatting and copying of selectors. #15999 #16165
  • [ENHANCEMENT] UI: Bring back vertical grid lines and graph legend series toggling instructions. #16163 #16164
  • [ENHANCEMENT] Mixin: The cluster label can be customized using clusterLabel. #15826
  • [PERF] TSDB: Optimize some operations on head chunks by taking shortcuts. #12659
  • [PERF] TSDB & Agent: Reduce memory footprint during WL replay. #15778
  • [PERF] Remote-Write: Reduce memory footprint during WAL replay. #16197
  • [PERF] API: Reduce memory footprint during header parsing. #16001
  • [PERF] Rules: Improve dependency evaluation, enabling better concurrency. #16039
  • [PERF] Scraping: Improve scraping performance for native histograms. #15731
  • [PERF] Scraping: Improve parsing of created timestamps. #16072
  • [BUGFIX] Scraping: Bump cache iteration after error to avoid false duplicate detections. #16174
  • [BUGFIX] Scraping: Skip native histograms series when ingestion is disabled. #16218
  • [BUGFIX] PromQL: Fix counter reset detection for native histograms. #15902 #15987
  • [BUGFIX] PromQL: Fix inconsistent behavior with an empty range. #15970
  • [BUGFIX] PromQL: Fix inconsistent annotation in quantile_over_time(). #16018
  • [BUGFIX] PromQL: Prevent label_join() from producing duplicates. #15975
  • [BUGFIX] PromQL: Ignore native histograms in scalar(), sort() and sort_desc(). #15964
  • [BUGFIX] PromQL: Fix annotations for binary operations between incompatible native histograms. #15895
  • [BUGFIX] Alerting: Consider alert relabeling when deciding whether alerts are dropped. #15979
  • [BUGFIX] Config: Set GoGC to the default value in case of an empty configuration. #16052
  • [BUGFIX] TSDB: Fix unknown series errors and potential data loss during WAL replay when inactive series are removed from the head and reappear before the next WAL checkpoint. #16060
  • [BUGFIX] Scaleway SD: The public IP will no longer be set to __meta_meta_scaleway_instance_public_ipv4 if it is an IPv6 address. #14228
  • [BUGFIX] UI: Display the correct value of Alerting rules' keep_firing_for. #16211

3.3.0-rc.1 / 2025-04-02

08 Apr 07:50
v3.3.0-rc.1
293f0c9
Compare
Choose a tag to compare
Pre-release
  • [BUGFIX] Remote-Write: Reduce memory footprint during WAL replay. #16197
  • [BUGFIX] Scraping: Skip native histograms series when ingestion is disabled. #16218
  • [BUGFIX] UI: Display the correct value of Alerting rules' keep_firing_for. #16211
  • [BUGFIX] PromQL: return NaN from irate() if second-last sample is NaN. #16199 #15853