Upgrade to Lucene 10.4 by rahulgoswami · Pull Request #4195 · apache/solr

rahulgoswami · 2026-03-07T05:45:59Z

https://issues.apache.org/jira/browse/SOLR-18143

Description

Upgrade Lucene dependency to 10.4

Solution

Followed instructions in dev-docs/lucene-upgrade.md and resolved compilation/test failures. Also made changes to documentation and upgrade guide wherever applicable.

rahulgoswami · 2026-03-07T05:50:18Z

WIP...fixing failures with DenseVectorField tests

…tSize_shouldThrowException

…to changes in Lucene104ScalarQuantizedVectorField format in Lucene 10.4

rahulgoswami · 2026-03-09T06:21:07Z

Put Claude(Opus 4.6) and Codex (GPT 5.4) to work on compilation errors and to understand test failures. Main test failures were around ScalarQuantizedDenseVectorField and BinaryQuantizedDenseVectorField due to breaking changes in Lucene104ScalarQuantizedVectorsFormat.

Major changes:

Lucene104HnswScalarQuantizedVectorsFormat has moved to an encoding based API. It no longer accepts "confidenceInterval" or "compression" params. Hence made those no-op in ScalarQuantizedDenseVectorField and removed the same from tests and documentation.
Added a note in documentation to say that older Solr 10.x schema may contain those params but not to be used going forward. Existing schemas will continue to be supported.
There is no separate binary quantization format in Lucene 10.4. Binary quantization is now just another encoding type of Lucene104ScalarQuantizedVectorsFormat (encoding=ScalarEncoding.SINGLE_BIT_QUERY_NIBBLE). But we'll need to continue to expose it at the Solr level as a separate type through BinaryQuantizedDenseVectorField for back compatibility.

rahulgoswami · 2026-03-09T06:28:32Z

Additionally, Lucene104ScalarQuantizedVectorsFormat now supports 1,2,4,7 and 8 bits in the format as opposed to only 4 and 7 earlier. Guess that should be scoped under a separate PR with test and documentation changes instead of squashing everything together in this upgrade PR.

rahulgoswami · 2026-03-09T06:44:42Z

@dsmiley @alessandrobenedetti Requesting a review please.

rahulgoswami · 2026-03-09T07:16:50Z

To-Do: Lucene104HnswScalarQuantizedVectorsFormat now defaults to 8 bits (ScalarEncoding.UNSIGNED_BYTE) instead of 7 bits in Lucene99HnswScalarQuantizedVectorsFormat earlier. Make that the new default for ScalarQuantizedDenseVectorField too (ScalarQuantizedDenseVectorField.DEFAULT_BITS)?

rahulgoswami · 2026-03-14T03:24:59Z

Also adding @liangkaiwen since he worked on these quantization types for 10.0. Kevin, could you please take a look at the related changes?

dsmiley

Remember solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc

dsmiley · 2026-03-15T13:44:49Z

dev-docs/lucene-upgrade.md

Remember the discussion thread about a Lucene upgrade that includes a Codec update, and needing to broadcast this fact in the upgrade notes page? This markdown file should explicitly add this step.

I don't think I remember a discussion thread regarding this. Quick search on the dev list wasn't helpful either. No mention of codec changes in the current solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc either. Do you have a link to the discussion ?

Here's the message I wrote in response to Jan's thread.

Ah that one! Added changes in major-changes-in-solr-10.adoc and lucene-upgrade.md to that effect. Thanks.

dsmiley · 2026-03-15T13:47:04Z

solr/core/src/java/org/apache/solr/schema/ScalarQuantizedDenseVectorField.java

+    // Retain parsing of these ("compress", "confidenceInterval") legacy Solr 10 schema params for
+    // compatibility even though
+    // Lucene 10.4's scalar-quantized vector format no longer consumes them directly. They are no-op
+    // going forward.


Then maybe lets mark the params deprecated with a version param of 10.1

Also, use the DeprecationLog so that the user sees a warning if they provide these params

cpoerschke · 2026-03-20T16:59:31Z

solr/core/src/java/org/apache/solr/schema/BinaryQuantizedDenseVectorField.java

-    return new Lucene102HnswBinaryQuantizedVectorsFormat(getHnswM(), getHnswEfConstruction());
+    return new Lucene104HnswScalarQuantizedVectorsFormat(
+        ScalarEncoding.SINGLE_BIT_QUERY_NIBBLE, getHnswM(), getHnswEfConstruction());


ref. apache/lucene#15353

cpoerschke · 2026-03-20T17:08:09Z

solr/core/src/java/org/apache/solr/schema/ScalarQuantizedDenseVectorField.java

-        useCompression(),
-        getConfidenceInterval(),


Perhaps reduce visibility of the accessors and/or mark them deprecated.

- public boolean useCompression() { + @Deprecated + @VisibleForTesting + boolean useCompression() {

cpoerschke · 2026-03-20T17:10:12Z

solr/solr-ref-guide/modules/query-guide/pages/dense-vector-search.adoc

-+
-Accepted values: `BOOLEAN`
+`NOTE:` Existing Solr 10 schemas may still contain legacy scalar-quantization parameters such as
+`confidenceInterval`, `dynamicConfidenceInterval`, or `compress`. They are accepted for backward


Suggested change

`confidenceInterval`, `dynamicConfidenceInterval`, or `compress`. They are accepted for backward

`confidenceInterval`, `dynamicConfidenceInterval`, or `compress`. They are deprecated (i.e. parsed and ignored but not used) for backward

…Log to notify the user

…d properties

rahulgoswami · 2026-03-22T06:19:06Z

Thank you for the review @dsmiley and @cpoerschke. I have incorporated all the comments except one by David (pending clarification).

…-solr-10.adoc in case of codec change

dsmiley · 2026-03-23T04:30:11Z

solr/solr-ref-guide/modules/query-guide/pages/dense-vector-search.adoc


 `NOTE:` Existing Solr 10 schemas may still contain legacy scalar-quantization parameters such as
-`confidenceInterval`, `dynamicConfidenceInterval`, or `compress`. They are accepted for backward
+`confidenceInterval`, `dynamicConfidenceInterval`, or `compress`. They are deprecated (i.e. parsed and ignored but not used) for backward


ignored implies not used.

I suggest slightly simpler wording:

They are now deprecated and ignored. New schemas should ...

You might even drop the "New schemas should..." sentence as it seems obvious (to me).

Oof! I completely botched the language there during a previous refactor. Fixed.

dsmiley · 2026-03-23T04:31:43Z

solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc

+
+Solr 10.1 upgrades the underlying Lucene library from 10.3 to 10.4, which introduces a new index codec (`Lucene104`).
+New index segments will be written using the `Lucene104` codec format.
+Existing segments written with older codecs (e.g. `Lucene103`) will continue to be readable.


Suggest simpler: Older segments will continue to be readable.

dsmiley · 2026-03-23T04:40:59Z

solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc

+
+WARNING: After upgrading to Solr 10.1, downgrading to an earlier Solr 10.0.x version may fail because the older version does not include the `Lucene104` codec needed to read the newly written segments.
+If you require the ability to roll back, back up your indexes before upgrading.
+A full reindex would be needed to downgrade after new segments have been written in Solr 10.1.


This sentence reads clearly yet I think it's message is confusing given what was said in the sentences before. For example, if we do backup-restore, why would a full reindex be needed? And you just said there may be errors for Solr 10 reading the indexes so, how would re-indexing (or any indexing or anything at all) work once you've seen such errors? You could drop this sentence entirely and I think we've already communicated what's needed.

I meant to communicate that IF there is no backup and you need to downgrade, a full reindex from source would be needed. Agree the missing "if no backup" pretext makes the sentence confusing. Will fix.

rahulgoswami added 2 commits March 6, 2026 02:15

upgrade Lucene to v10.4.0

809ef97

Fix PostingsFormat test failure

c4a7124

rahulgoswami marked this pull request as draft March 7, 2026 05:47

rahulgoswami added 3 commits March 7, 2026 19:26

Fix ScalarQuantizedDenseVectorFieldTest.fieldTypeDefinition_invalidBi…

04b4190

…tSize_shouldThrowException

Changes for ScalarQuantizedVectorField (tests and documentation) due …

37550e0

…to changes in Lucene104ScalarQuantizedVectorField format in Lucene 10.4

changelog

e7c49e2

rahulgoswami marked this pull request as ready for review March 9, 2026 06:21

rahulgoswami requested a review from dsmiley March 9, 2026 06:21

rahulgoswami requested a review from alessandrobenedetti March 9, 2026 06:22

dsmiley reviewed Mar 15, 2026

View reviewed changes

cpoerschke reviewed Mar 20, 2026

View reviewed changes

rahulgoswami added 2 commits March 22, 2026 02:10

Deprecate properties now unsupported in Lucene 10.4 and add Deprecate…

576b18e

…Log to notify the user

re-word deprecation notice in dense-vector-search.adoc for unsupporte…

29d3a64

…d properties

rahulgoswami added 2 commits March 22, 2026 23:38

Add warning about codec change restricting downgradability in Solr 10.1

a7c87bb

Add a step in lucene-upgrade.md to include a note in major-changes-in…

ca85977

…-solr-10.adoc in case of codec change

rahulgoswami requested review from cpoerschke and dsmiley March 23, 2026 04:15

dsmiley reviewed Mar 23, 2026

View reviewed changes

Simplification of language in ref-guide

81b268a

	`confidenceInterval`, `dynamicConfidenceInterval`, or `compress`. They are accepted for backward
	`confidenceInterval`, `dynamicConfidenceInterval`, or `compress`. They are deprecated (i.e. parsed and ignored but not used) for backward

Conversation

rahulgoswami commented Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Solution

Uh oh!

rahulgoswami commented Mar 7, 2026

Uh oh!

rahulgoswami commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rahulgoswami commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rahulgoswami commented Mar 9, 2026

Uh oh!

rahulgoswami commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rahulgoswami commented Mar 14, 2026

Uh oh!

dsmiley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgoswami Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgoswami Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgoswami commented Mar 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgoswami Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rahulgoswami commented Mar 7, 2026 •

edited

Loading

rahulgoswami commented Mar 9, 2026 •

edited

Loading

rahulgoswami commented Mar 9, 2026 •

edited

Loading

rahulgoswami commented Mar 9, 2026 •

edited

Loading

rahulgoswami Mar 22, 2026 •

edited

Loading

rahulgoswami Mar 23, 2026 •

edited

Loading

rahulgoswami Mar 23, 2026 •

edited

Loading