Releases: prometheus/prometheus
Releases · prometheus/prometheus
3.5.0-rc.1 / 2025-07-08
3.5 will be a Long Term Support (LTS) release; see release https://prometheus.io/docs/introduction/release-cycle/
There are a number of new features, and no known breaking changes in this release.
The following feature was removed since RC.1, since problems were discovered:
- [FEATURE] OTLP: Support promoting OTel scope name/version/schema URL/attributes as metric labels, via
otlp.promote_scope_metadata
. #16730 #16760
What's Changed
- [FEATURE] PromQL: Add experimental type and unit metadata labels, behind feature flag
type-and-unit-labels
. #16228 #16632 #16718 #16743 - [FEATURE] PromQL: Add
ts_of_(min|max|last)_over_time
, behind feature flagexperimental-promql-functions
. #16722 #16733 - [FEATURE] Scraping: Add global option
always_scrape_classic_histograms
to scrape a classic histogram even if it is also exposed as native. #16452 - [FEATURE] OTLP: New config options
promote_all_resource_attributes
andignore_resource_attributes
. #16426 - [FEATURE] Discovery: New service discovery for STACKIT Cloud. #16401
- [ENHANCEMENT] Hetzner SD: Add
label_selector
to filter servers. #16512 - [ENHANCEMENT] PromQL: support non-constant parameter in aggregations like
quantile
andtopk
. #16404 - [ENHANCEMENT] UI: Better total target count display when using
keep_dropped_targets
option. #16604 - [ENHANCEMENT] UI: Add simple filtering on the
/rules
page. #16605 - [ENHANCEMENT] UI: Display query stats in hover tooltip over table query tab. #16723
- [ENHANCEMENT] UI: Clear search field on
/targets
page. #16567 - [ENHANCEMENT] Rules: Check that rules parse without error earlier at startup. #16601
- [ENHANCEMENT] Promtool: Optional fuzzy float64 comparison in rules unittests. #16395
- [PERF] PromQL: Reuse
histogramStatsIterator
where possible. #16686 - [PERF] PromQL: Reuse storage for custom bucket values for native histograms. #16565
- [PERF] UI: Optimize memoization and search debouncing on
/targets
page. #16589 - [PERF] UI: Fix full-page re-rendering when opening status nav menu. #16590
- [PERF] Kubernetes SD: use service cache.Indexer to achieve better performance. #16365
- [PERF] TSDB: Optionally use Direct IO for chunks writing. #15365
- [PERF] TSDB: When fetching label values, stop work earlier if the limit is reached. #16158
- [PERF] Labels: Simpler/faster stringlabels encoding. #16069
- [PERF] Scraping: Reload scrape pools concurrently. #16595 #16783
- [BUGFIX] Top-level: Update GOGC before loading TSDB. #16491
- [BUGFIX] Config: Respect GOGC environment variable if no "runtime" block exists. #16558
- [BUGFIX] PromQL: Fix native histogram
last_over_time
. #16744 - [BUGFIX] PromQL: Fix reported parser position range in errors for aggregations wrapped in ParenExpr #16041 #16754
- [BUGFIX] PromQL: Don't emit a value from
histogram_fraction
orhistogram_quantile
if classic and native histograms are present at the same timestamp. #16552 - [BUGFIX] PromQL: Incorrect rounding of
[1001ms]
to[1s]
and similar. #16478 - [BUGFIX] PromQL: Fix inconsistent / sometimes negative
histogram_count
andhistogram_sum
. #16682 - [BUGFIX] PromQL: Improve handling of NaNs in native histograms. #16724
- [BUGFIX] PromQL: Fix unary operator precedence in duration expressions. #16713
- [BUGFIX] PromQL: Improve consistency of
avg
aggregation andavg_over_time
. #16569 #16773 - [BUGFIX] UI: Add query warnings and info to graph view. #16753 #16759
- [BUGFIX] API: Add HTTP
Vary: Origin
header to responses to avoid cache poisoning. #16008 - [BUGFIX] Discovery: Avoid deadlocks by taking locks in consistent order. #16587
- [BUGFIX] Remote-write: For Azure AD auth, allow empty
client_id
to suppport system assigned managed identity. #16421 - [BUGFIX] Scraping: Fix rare memory corruption bug. #16623
- [BUGFIX] Scraping: continue handling custom-bucket histograms after an exponential histogram is encountered. #16720
- [BUGFIX] OTLP: Default config not respected when
otlp:
block is unset. #16693
3.5.0-rc.0 / 2025-06-25
3.5 will be a Long Term Support (LTS) release; see release https://prometheus.io/docs/introduction/release-cycle/
There are a number of new features, and no known breaking changes in this release.
What's Changed
- [FEATURE] PromQL: Add experimental type and unit metadata labels, behind feature flag
type-and-unit-labels
. #16228 #16632 #16718 #16743 - [FEATURE] PromQL: Add
ts_of_(min|max|last)_over_time
, behind feature flagexperimental-promql-functions
. #16722 #16733 - [FEATURE] Scraping: Add global option
always_scrape_classic_histograms
to scrape a classic histogram even if it is also exposed as native. #16452 - [FEATURE] OTLP: Support promoting OTel scope name/version/schema URL/attributes as metric labels, via
otlp.promote_scope_metadata
. #16730 #16760 - [FEATURE] OTLP: New config options
promote_all_resource_attributes
andignore_resource_attributes
. #16426 - [FEATURE] Discovery: New service discovery for STACKIT Cloud. #16401
- [ENHANCEMENT] Hetzner SD: Add
label_selector
to filter servers. #16512 - [ENHANCEMENT] PromQL: support non-constant parameter in aggregations like
quantile
andtopk
. #16404 - [ENHANCEMENT] UI: Better total target count display when using
keep_dropped_targets
option. #16604 - [ENHANCEMENT] UI: Add simple filtering on the
/rules
page. #16605 - [ENHANCEMENT] UI: Display query stats in hover tooltip over table query tab. #16723
- [ENHANCEMENT] UI: Clear search field on
/targets
page. #16567 - [ENHANCEMENT] Rules: Check that rules parse without error earlier at startup. #16601
- [ENHANCEMENT] Promtool: Optional fuzzy float64 comparison in rules unittests. #16395
- [PERF] PromQL: Reuse
histogramStatsIterator
where possible. #16686 - [PERF] PromQL: Reuse storage for custom bucket values for native histograms. #16565
- [PERF] UI: Optimize memoization and search debouncing on
/targets
page. #16589 - [PERF] UI: Fix full-page re-rendering when opening status nav menu. #16590
- [PERF] Kubernetes SD: use service cache.Indexer to achieve better performance. #16365
- [PERF] TSDB: Optionally use Direct IO for chunks writing. #15365
- [PERF] TSDB: When fetching label values, stop work earlier if the limit is reached. #16158
- [PERF] Labels: Simpler/faster stringlabels encoding. #16069
- [PERF] Scraping: Reload scrape pools concurrently. #16595 #16783
- [BUGFIX] Top-level: Update GOGC before loading TSDB. #16491
- [BUGFIX] Config: Respect GOGC environment variable if no "runtime" block exists. #16558
- [BUGFIX] PromQL: Fix native histogram
last_over_time
. #16744 - [BUGFIX] PromQL: Fix reported parser position range in errors for aggregations wrapped in ParenExpr #16041 #16754
- [BUGFIX] PromQL: Don't emit a value from
histogram_fraction
orhistogram_quantile
if classic and native histograms are present at the same timestamp. #16552 - [BUGFIX] PromQL: Incorrect rounding of
[1001ms]
to[1s]
and similar. #16478 - [BUGFIX] PromQL: Fix inconsistent / sometimes negative
histogram_count
andhistogram_sum
. #16682 - [BUGFIX] PromQL: Improve handling of NaNs in native histograms. #16724
- [BUGFIX] PromQL: Fix unary operator precedence in duration expressions. #16713
- [BUGFIX] PromQL: Improve consistency of
avg
aggregation andavg_over_time
. #16569 #16773 - [BUGFIX] UI: Add query warnings and info to graph view. #16753 #16759
- [BUGFIX] API: Add HTTP
Vary: Origin
header to responses to avoid cache poisoning. #16008 - [BUGFIX] Discovery: Avoid deadlocks by taking locks in consistent order. #16587
- [BUGFIX] Remote-write: For Azure AD auth, allow empty
client_id
to suppport system assigned managed identity. #16421 - [BUGFIX] Scraping: Fix rare memory corruption bug. #16623
- [BUGFIX] Scraping: continue handling custom-bucket histograms after an exponential histogram is encountered. #16720
- [BUGFIX] OTLP: Default config not respected when
otlp:
block is unset. #16693
2.53.5 / 2025-06-27
3.4.2 / 2025-06-26
3.4.1 / 2025-05-31
- [BUGFIX] Parser: Add reproducer for a dangling-reference issue in parsers. #16633
3.4.0 / 2025-05-17
- [CHANGE] Config: Make setting out-of-order native histograms feature (
--enable-feature=ooo-native-histograms
) a no-op. Out-of-order native histograms are now always enabled whenout_of_order_time_window
is greater than zero and--enable-feature=native-histograms
is set. #16207 - [FEATURE] OTLP translate: Add feature flag for optionally translating OTel explicit bucket histograms into native histograms with custom buckets. #15850
- [FEATURE] OTLP translate: Add option to receive OTLP metrics without translating names or attributes. #16441
- [FEATURE] PromQL: allow arithmetic operations in durations in PromQL parser. #16249
- [FEATURE] OTLP receiver: Add primitive support for ingesting OTLP delta metrics as-is. #16360
- [ENHANCEMENT] PromQL: histogram_fraction for bucket histograms. #16095
- [ENHANCEMENT] TSDB: add
prometheus_tsdb_wal_replay_unknown_refs_total
andprometheus_tsdb_wbl_replay_unknown_refs_total
metrics to track unknown series references during WAL/WBL replay. #16166 - [ENHANCEMENT] Scraping: Add config option for escaping scheme request. #16066
- [ENHANCEMENT] Config: Add global config option for convert_classic_histograms_to_nhcb. #16226
- [ENHANCEMENT] Alerting: make batch size configurable (
--alertmanager.notification-batch-size
). #16254 - [PERF] Kubernetes SD: make endpointSlice discovery more efficient. #16433
- [BUGFIX] Config: Fix auto-reload on changes to rule and scrape config files. #16340
- [BUGFIX] Scraping: Skip native histogram series if ingestion is disabled. #16218
- [BUGFIX] TSDB: Handle metadata/tombstones/exemplars for duplicate series during WAL replay. #16231
- [BUGFIX] TSDB: Avoid processing exemplars outside the valid time range during WAL replay. #16242
- [BUGFIX] Promtool: Add feature flags for PromQL features. #16443
- [BUGFIX] Rules: correct logging of alert name & template data. #15093
- [BUGFIX] PromQL: Use arithmetic mean for
histogram_stddev()
andhistogram_stdvar()
. #16444
3.4.0-rc.0 / 2025-04-29
- [CHANGE] Config: Make setting out-of-order native histograms feature (
--enable-feature=ooo-native-histograms
) a no-op. Out-of-order native histograms are now always enabled whenout_of_order_time_window
is greater than zero and--enable-feature=native-histograms
is set. #16207 - [FEATURE] OTLP translate: Add feature flag for optionally translating OTel explicit bucket histograms into native histograms with custom buckets. #15850
- [FEATURE] OTLP translate: Add option to receive OTLP metrics without translating names or attributes. #16441
- [FEATURE] PromQL: allow arithmetic operations in durations in PromQL parser. #16249
- [FEATURE] OTLP receiver: Add primitive support for ingesting OTLP delta metrics as-is. #16360
- [ENHANCEMENT] PromQL: histogram_fraction for bucket histograms. #16095
- [ENHANCEMENT] TSDB: add
prometheus_tsdb_wal_replay_unknown_refs_total
andprometheus_tsdb_wbl_replay_unknown_refs_total
metrics to track unknown series references during WAL/WBL replay. #16166 - [ENHANCEMENT] Scraping: Add config option for escaping scheme request. #16066
- [ENHANCEMENT] Config: Add global config option for convert_classic_histograms_to_nhcb. #16226
- [ENHANCEMENT] Alerting: make batch size configurable (
--alertmanager.notification-batch-size
). #16254 - [PERF] Kubernetes SD: make endpointSlice discovery more efficient. #16433
- [BUGFIX] Config: Fix auto-reload on changes to rule and scrape config files. #16340
- [BUGFIX] Scraping: Skip native histogram series if ingestion is disabled. #16218
- [BUGFIX] TSDB: Handle metadata/tombstones/exemplars for duplicate series during WAL replay. #16231
- [BUGFIX] TSDB: Avoid processing exemplars outside the valid time range during WAL replay. #16242
- [BUGFIX] Promtool: Add feature flags for PromQL features. #16443
- [BUGFIX] Rules: correct logging of alert name & template data. #15093
- [BUGFIX] PromQL: Use arithmetic mean for
histogram_stddev()
andhistogram_stdvar()
. #16444
3.3.1 / 2025-05-02
3.3.0 / 2025-04-15
- [FEATURE] PromQL: Implement
idelta()
andirate()
for native histograms. #15853 - [ENHANCEMENT] Scaleway SD: Add
__meta_scaleway_instance_public_ipv4_addresses
and__meta_scaleway_instance_public_ipv6_addresses
labels. #14228 - [ENHANCEMENT] TSDB: Reduce locking while reloading blocks. #12920
- [ENHANCEMENT] PromQL: Allow UTF-8 labels in
label_replace()
. #15974 - [ENHANCEMENT] Promtool:
tsdb create-blocks-from openmetrics
can now read from a Pipe. #16011 - [ENHANCEMENT] Rules: Add support for anchors and aliases in rule files. #14957
- [ENHANCEMENT] Dockerfile: Make
/prometheus
writable. #16073 - [ENHANCEMENT] API: Include scrape pool name for dropped targets in
/api/v1/targets
. #16085 - [ENHANCEMENT] UI: Improve time formatting and copying of selectors. #15999 #16165
- [ENHANCEMENT] UI: Bring back vertical grid lines and graph legend series toggling instructions. #16163 #16164
- [ENHANCEMENT] Mixin: The
cluster
label can be customized usingclusterLabel
. #15826 - [PERF] TSDB: Optimize some operations on head chunks by taking shortcuts. #12659
- [PERF] TSDB & Agent: Reduce memory footprint during WL replay. #15778
- [PERF] Remote-Write: Reduce memory footprint during WAL replay. #16197
- [PERF] API: Reduce memory footprint during header parsing. #16001
- [PERF] Rules: Improve dependency evaluation, enabling better concurrency. #16039
- [PERF] Scraping: Improve scraping performance for native histograms. #15731
- [PERF] Scraping: Improve parsing of created timestamps. #16072
- [BUGFIX] Scraping: Bump cache iteration after error to avoid false duplicate detections. #16174
- [BUGFIX] Scraping: Skip native histograms series when ingestion is disabled. #16218
- [BUGFIX] PromQL: Fix counter reset detection for native histograms. #15902 #15987
- [BUGFIX] PromQL: Fix inconsistent behavior with an empty range. #15970
- [BUGFIX] PromQL: Fix inconsistent annotation in
quantile_over_time()
. #16018 - [BUGFIX] PromQL: Prevent
label_join()
from producing duplicates. #15975 - [BUGFIX] PromQL: Ignore native histograms in
scalar()
,sort()
andsort_desc()
. #15964 - [BUGFIX] PromQL: Fix annotations for binary operations between incompatible native histograms. #15895
- [BUGFIX] Alerting: Consider alert relabeling when deciding whether alerts are dropped. #15979
- [BUGFIX] Config: Set
GoGC
to the default value in case of an empty configuration. #16052 - [BUGFIX] TSDB: Fix unknown series errors and potential data loss during WAL replay when inactive series are removed from the head and reappear before the next WAL checkpoint. #16060
- [BUGFIX] Scaleway SD: The public IP will no longer be set to
__meta_meta_scaleway_instance_public_ipv4
if it is an IPv6 address. #14228 - [BUGFIX] UI: Display the correct value of Alerting rules'
keep_firing_for
. #16211
3.3.0-rc.1 / 2025-04-02
- [BUGFIX] Remote-Write: Reduce memory footprint during WAL replay. #16197
- [BUGFIX] Scraping: Skip native histograms series when ingestion is disabled. #16218
- [BUGFIX] UI: Display the correct value of Alerting rules'
keep_firing_for
. #16211 - [BUGFIX] PromQL: return NaN from
irate()
if second-last sample is NaN. #16199 #15853