1.1. 3.5.x Branch

1.1.1. Version 3.5.3 

Note

Erlang/OTP 27 is the minimum supported version

1.1.1.1. Features

#6069, #6143: New replicator authentication plugin couch_replicator_auth_ibm for IBM IAM API keys. Disabled by default. Enabled through [replicator] auth_plugins.

1.1.1.2. Performance

#6077: Remove a quadratic scan in the _bulk_docs hot path in couch_db_updater.
#6018, #6021, #6023: QuickJS engine updates. Custom small-block allocator (10-40% faster in microbenchmarks), fix for a large performance regression with recent GCC versions, faster math ops.
#5883: Use the BTree term cache for view files as well as database files, which improves latency of concurrent view queries. Adds the [bt_engine_cache] view_btree_cache_depth setting.
#5890: Use hibernate_after instead of hibernating after every message in couch_work_queue, couch_db_updater, couch_stream and rexi_buffer. Adds the [hibernate_after] section to configure or disable idle timeouts.
#6115: Optimize and improve validation of changes sequence decoding. Decoding is 2-3x faster. The improvements apply to search (Dreyfus) bookmarks as well.

1.1.1.3. Bugfixes

#6004, #6005: Fix _bulk_get not_found handling when revs=all and a node is in maintenance mode, and accumulate worker errors properly.
#6016: Fix a leaked database handle in smoosh, which kept deleted database files open.
#6025: fsync after resetting or truncating file headers.
#6026: Always apply the user-configured file_compression setting. Purge trees and the compactor’s external sort used snappy regardless of the setting.
#6022, #6024: Limit couch_server sharding with the new [couchdb] max_couch_server_shards setting so that max_dbs_open is not split into tiny per-shard limits on many-core nodes. Add an all_dbs_active metric.
#6043: Fix function_clause in fabric_ring for operations using the all ring option (_uuids, _purged_infos, db metadata updates) when a node is in maintenance mode.
#6045: Return 400 Bad Request instead of 500 for invalid revisions passed to _missing_revs, _revs_diff and _purge.
#6089: Validate limit and timeout parameters to _changes and return 400 instead of 500.
#6090: Handle badarg from base64:decode when parsing authorization headers.
#6061: Improve JWT error messages.
#6066: Assume a purge client exists when its existence cannot be verified, in view, Dreyfus and Nouveau purge checkpoints.
#6130: Preserve view purge sequences when compacting. No longer skip purges that happen right before a view is compacted.
#6140: Fix purging non-leaf revisions in map-reduce views.
#6129: Fix attachment revision positions when re-creating a deleted document. Attachments re-created with the same name will now replicate from source to target when previously they weren’t.
#6081: Handle noconnection, timeouts and exits from remote nodes during index cleanup.
#6074: Handle empty design documents in the ddoc_features scanner plugin.
#6100: Fix a variable shadowing typo in the mem3 reshard source shard check.
#6102: Fix group_by_proximity returning an empty same-zone shard list when the coordinator does not hold a copy of the shard.
#6145: Fix spurious 500 during concurrent database deletion.
#5948: Handle {bad_request, Error, Reason} triples from design document validators in fabric_doc_update.
#5839: Only increment the vdu_rejects counter on actual VDU rejections. Only this fix from that pull request is included.
#6056: Nouveau: always use cacertfile when specified and make peer verification explicit. Adds the [nouveau] ssl_verify setting.
#6078: Nouveau: fix explicit sorting by relevance (-<score>).
#6067: Nouveau: fix a typo in the Portuguese analyzer name and remove unused query serializer classes.
#6096: Nouveau: close analyzers after use and use the default value in all cases in the index manager.
#6125: Nouveau: fix string sorting order to match Java’s lexicographic order.
#6138: Nouveau: fix purge out of order index update
#6018, #6021, #6023, #6133, #6156: QuickJS updates.
#6058: Update Jiffy to 2.0.2
#6037, #6082: Update Gun to 2.5.0.
#6065: New Prometheus metrics for Clouseau connectivity, database count, IOQ and Smoosh.
#6068: Add a query server process_count gauge.
#6070: Supervisor-level replicator auth plugin callbacks sup_initialize/0 and sup_cleanup/1.
#6141: Limit number of redirects in replicator.
#6147: Make _search_cleanup and _nouveau_cleanup admin-only like _view_cleanup.
#6149: Fix the _users and _replicator database name suffix check in the auth handler.
#6148: Fix default proxy authentication headers config value.
#6151: Use incremental decompression for reques bodies
#6152: Use short names for auth handlers. The previous tuple forms are still accepted.
#6153: Improve command running in weatherreport
#6134: Fix the scanner backoff logic.
#6157: Strip credentials from replication documents read by non-owners in a shared _replicator database, from log messages and from replication status responses.
#6155: Update Fauxton to v1.3.7.

1.1.1.4. Docs

#6047, #6049: Update RHEL 10 install instructions.
#6099: Update Search (Clouseau) installation instructions.
#6055: Clarify Nouveau lucene_version handling.
#6057: Fix a JavaScript for loop in the partitioned dbs example.
#6062: Fix the revision label in the consistency introduction.
#6089: Remove an erroneous comment about _changes limit=0.
#6094: Fix an inline link in the auth configuration docs.
#6097: Update the build status link in README.rst.

1.1.1.5. Tests/CI/Builds

#6009, #6036, #6038: Set OTP 27 minimum version.
#6027, #6035, #6037, #6083, #6095: Update dependencies: mochiweb, snappy, ibrowse, meck.
#6007: Update Gradle to 9.5.1 for Nouveau.
#6017, #6019, #6021, #6073, #6091, #6093, #5990, #6112, #6131, #6123: CI updates: match variants to worker count, move base images to Trixie, adjust Windows settings, update Erlang patch versions, skip PPC64LE for now.
#6033, #6042, #6084: Test reliability fixes.
#6076: Use OTP 27 sigils in chttpd_db_doc_size_tests.
#6135: dev/run escalates to SIGKILL when a Clouseau process ignores SIGTERM during teardown.
#6150: Windows: configure.ps1 extracts clouseau.zip flat, as needed since Clouseau 3.4.0.

Shortly after the release of CouchDB 3.5.2, the Erlang team released a new version of Erlang/OTP 26 containing security fixes. We have therefore decided to update our convenience binaries as well, to use the updated Erlang/OTP version. For some platforms, these are being released as CouchDB 3.5.2.1.

1.1.2.1. Features

#5628, #5940: Nouveau Lucene 10 upgrade. Lucene is bumped to 10.4.0. New Nouveau indexes use Lucene 10 by default. Existing Lucene 9 indexes continue to work via version negotiation, and an upgrade scanner (nouveau_index_upgrader) can rebuild them at a configurable rate.
#5896: Nouveau bulk index updates. Replace doc-at-a-time updates with bulk updates which should improve index building.
#5986: Fix cluster index cleanup. Adds the [couchdb] index_cleanup_delay_msec setting.
#5624: Use SpiderMonkey 128 as the default version.
#5754: Support safe secret rotation.
#5820: Avoid updating password hash when request uses simple password scheme.

1.1.2.2. Performance

#5982: Update Jiffy to 2.0. Jiffy 2.0 has a number of performance optimizations which should speed up both encoding and decoding.
#5994: Optimize replicator. Use gen_server casts to queue docs for _bulk_docs in replication workers instead of synchronous calls.
#5987: Do not run a full GC after each index doc update. Increase view queue size from 100KB to 10MB. This should improve view index building times.

1.1.2.3. Bugfixes

#5751, #5759, #5795, #5836, #5937: QuickJS engine updates. Memory leak fixes, use-after-free fixes in workers and regex, regex memory blowup fix, Unicode 17 support, FP16 support, optimized string operations, faster context creation, closure optimization, iterator improvements.
#5746, #5748, #5882, #5941: QuickJS scanner plugin and dispatch fixes. Reset docs_size in the scanner. Add a skip option to the doc_fdi callback. Simplify the scanner plugin. Fix ReferenceError in QuickJS dispatch for unknown commands.
#5758: Fix purge mem3_rep client verification.
#5827: Fix race condition during purge checkpoint creation.
#5832, #5833: Fix handling shards dbs purge checkpoints in mem3_rep. Consider previous-node replications for _dbs purge checkpoints.
#5912, #5913, #5918: Fix signatures in mrview purge checkpoints. Avoid creating purge checkpoints for invalid views. Do not reset dreyfus purge_seq back to 0.
#5940: Initialise nouveau index purge_seq from db purge_seq, and commit purge seq update before returning.
#5931: Handle cases when dreyfus checkpoint is out-of-sync with the index.
#5939: Fix view compaction duration logging.
#5777: Fix replicator scheduler total jobs metric.
#5793: Fix Prometheus metrics annotations.
#5881: Improve replication since_seq parameter handling.
#5876: Fix intermittent _scheduler/docs 500 error.
#5853: Tolerate invalid typ claim when not required.
#5846: Tolerate maintenance mode and a node down in partition info calls.
#5807: Fix bulk_get error handling.
#5978: Fix _find when over threshold.
#5976: Use errstr() instead of toSource() in nouveau JS wrapper.
#5893: Send 404 for /_all_dbs and /_dbs_info with extra path parts.

1.1.2.4. Docs

#5973: Fix Jenkins badge link in README.
#5974: Update CI badge URL in README.rst.
#5992: Add missing comment from docs.
#5875: Fix docs about replication.
#5925: Change CouchDB links to use HTTPS.
#5924: Update URLs to use HTTPS in mac installation guide.

1.1.2.5. Tests/CI/Builds

#5808, #5847, #5914: CI parallelism. Parallel make eunit (sub-targets / -j4 / Windows). Concat eunit test logs and remove tmp dirs in Makefile. Speed up compiling and release building.
#5971: Update Black to 26.3.1.
#5687, #5964, #5967, #5968: couch.uri improvements. Write couch_uri files so ports are discoverable with --auto-ports. Use couchdb.uri only when auto-ports is enabled. Make file path for couch.uri absolute. Make creation of couch.uri independent of --auto-ports. Clean up the couch.uri on make devclean.
#5864: Enable EUnit/Elixir/Nouveau tests on Windows.
#5933, #5936, #5950: Improve / fix Jenkins workspace cleanup.
#5946: CI: enable xref checking for undefined functions.
#5813: Don’t run tests if we only change .github configuration.
#5897: Temporarily disable freebsd-arm worker.
#5935: Improve CI performance by combining some stages.
#5665: Add Trixie to CI.

1.1.3. Version 3.5.1 

1.1.3.1. Features

#5626, #5665: Debian Trixie support
#5709: Automatic Nouveau and Clouseau index cleanup
#5697: Add UUID v7 as a uuid algorithm option. The default is still the default sequential algorithm.
#5713, #5697, #5701, #5704: Purge improvements and fixes. Optimize it up to ~30% faster for large batches. max_document_id_number setting was removed and max_revisions_number set to unlimited by default to match _bulk_docs and _bulk_get endpoints.
#5611: Implement the ability to downgrade CouchDB versions
#5588: Populate zone from COUCHDB_ZONE env variable in Docker
#5563: Set Erlang/OTP 26 as minimum supported version
#5546, #5641: Improve Clouseau service checks in clouseau_rpc module.
#5639: Use OS certificates for replication
#5728: Configurable reduce limit threshold and ratio

1.1.3.2. Performance

#5625: BTree engine term cache
#5617: Optimize Nouveau searches when index is fresh
#5598: Use HTTP/2 for Nouveau
#5701: Optimize revid parsing: 50-90% faster. Should help purge requests as well as _bulk_docs and _bulk_get endpoints.
#5564: Use the built-in binary hex encode
#5613: Improve scanner performance
#5545: Bump process limit to 1M

1.1.3.3. Bugfixes

#5722, #5683, #5678, #5646, #5630, #5615, #5696: Scanner fixes. Add write limiting and switch to traversing documents by sequence IDs instead of by document IDs.
#5707, #5706, #5706, #5694, #5691, #5669, #5629, #5574, #5573, #5566, #5553, #5550, #5534, #5730: QuickJS Updates. Optimized string operations, faster context creation, a lot of bug fixes.
#5719: Use “all” ring options for purged_infos
#5649: Retry call to dreyfus index on noproc errors
#5663: More informative error if epochs out of order
#5649: Dreyfus retries on error
#5643: Fix reduce_limit = log feature
#5620: Use copy_props in the compactor instead of set_props
#5632, #5627, #5607: Nouveau fixes. Enhance _nouveau_cleanup. Improve security on http/2.
#5614: Stop replication jobs to nodes which are not part of the cluster
#5596: Fix query args parsing during cluster upgrades
#5595: Make replicator shutdown a bit more orderly
#5595: Avoid making a mess in the logs when stopping replicator app
#5588: Fix couch_util:set_value/3
#5587: Improve mem3_rep:find_source_seq/4 logging
#5586: Don’t wait indefinitely for replication jobs to stop
#5578: Use [sync] option in couch_bt_engine:commit_data/1
#5556: Add guards to fabric:design_docs/1 to prevent function_clause error
#5555: Improve replicator client mailbox flush
#5551: Handle bad_generator and case_clause in ken_server
#5552: Improve cluster startup logging
#5552: Improve mem3 supervisor
#5552: Handle shard opener tables not being initializes better
#5549: Don’t spawn more than one init_delete_dir instance
#5535: Disk monitor always allows mem3_rep checkpoints
#5536: Fix mem3_util overlapping shards
#5533: No cfile support for 32bit systems
#5688: Handle timeout in dreyfus_fabric_search
#5548: Fix config key typo in mem3_reshard_dbdoc
#5540: Ignore extraneous cookie in replicator session plugin

1.1.3.4. Cleanups

#5717: Do not check for Dreyfus. It’s part of the tree now.
#5715: Remove Hastings references
#5714: Cleanup fabric r/w parameter handling
#5693: Remove explicit erlang module prefix for auto-imported functions
#5686: Remove erlang: prefix from erlang:error()
#5686: Fix case_clause when got missing_target error
#5690: Fix props caching in mem3
#5680: Implement db doc updating
#5666: Replace gen_server:format_status/2 with format_status/1
#5672: Cache and store mem3 shard properties in one place only
#5644: Remove redundant *_to_list / list_to_* conversion
#5633: Use config:get_integer/3 in couch_btree
#5618: DRY out couch_bt_engine header pointer term access
#5614: Stop replication jobs to nodes which are not part of the cluster
#5610: Add a range_to_hex/1 utility function
#5565: Use maps comprehensions and generators in a few places
#5649: Remove pointless message
#5649: Remove obsolete clauses from dreyfus
#5621: Minor couch_btree refactoring

1.1.3.5. Docs

#5705: Docs: Update the /_up endpoint docs to include status responses
#5653: Document that _all_dbs endpoint supports inclusive_end query param
#5575: Document how to mitigate high memory usage in docker
#5600: Avoid “master” wording at setup cluster
#5381: Change unauthorized example to 401 for replication
#5682: Update install instructions
#5674: Add setup documentation for two factor authentication
#5562: Add AI policy
#5548: Fix reshard doc section name
#5543: Add https to allowed replication proxy protocols

1.1.3.6. Tests/CI/Builds

#5720: Update deps: Fauxton, meck and PropEr
#5708: Improve search test
#5702: Increase timeout for process_response/3 to fix flaky tests
#5703: Use deterministic doc IDs in Mango key test
#5692: Implement ‘assert_on_status’ macro
#5684: Sequester docker ARM builds and fail early
#5679: Add --disable-spidermonkey to --dev[-with-nouveau]
#5671: Print request/response body on errors from mango test suite
#5670: Fix make clean after dev/run --enable-tls
#5668: Update xxHash
#5667: Update mochiweb to v3.3.0
#5664: Disable ppc64le and s390x builds
#5604: Use ASF fork of gun for cowlib dependency
#5636: Reduce btree prop test count a bit
#5633: Fix and improve couch_btree testing
#5572: Remove a few more instances of Ubuntu Focal
#5571: Upgrade Erlang for CI
#5570: Skip macos CI for now and remove Ubuntu Focal
#5488: Bump Clouseau to 2.25.0
#5541: Enable Clouseau for the Windows CI
#5537: Add retries to native full CI stage
#5531: Fix Erlang cookie configuration in dev/run
#5662: Remove old Jenkinsfiles
#5661: Unify CI jobs

1.1.4. Version 3.5.0 

1.1.4.1. Highlights

#5399, #5441, #5443, #5460, #5461: Implement parallel pread calls: lets clients issue concurrent pread calls without blocking each other or having to wait for all writes and fsync calls. This is enabled by default and can be disabled with [couchdb] use_cfile = false in the configuration.

CouchDB already employs a multiple-parallel-read and concurrent serial-write design at the database engine layer, but below that in the storage engine, each file representing a database shard is required to route all read / write / sync requests through a single Erlang process that owns the file descriptor (fd).

An Erlang process can at most execute at the speed of a single CPU core. Two scenarios can lead to a starvation of the fd-owning Erlang process message inbox:
- 1000s of concurrent read requests with a constant stream of writes per shard, or:
- a high latency storage devices like network block storage.
Parallel preads re-implements the Erlang file module in parts as couch_cfile to allow multiple dup()’d file descriptors to be used for reading and writing. Writes continue to be serialised at the database engine layer, but reads now are no longer blocked by writes waiting to commit.

Comes with an extensive test suite that includes property testing to ensure couch_cfile behaves exactly like Erlang’s file in all other cases.

Performance is always equal or better than before. These scenarios show preliminary improvements:
- random document reads: 15% more throughput
- read _all_docs: 15% more throughput
- read _all_docs with include_docs=true: 25% more throughput
- read _changes: 4% more throughput
- single document writes: 8% more throughput
- 2000x concurrent clients, random document reads on a 12 node cluster: 30% more throughput
- concurrent constant document writes and concurrent single document reads on a single shard: 40% more throughput.
  
  This feature is not available on Windows.
#5435: Improve default chttpd_server options. This helps with faster processing of many concurrent TCP connections.
#5347: Fix attachment size calculation. This could lead to shards not being scheduled for compaction correctly.
#5494: Implement _top_N and _bottom_N reducers. These reducers return the top or bottom values from a map-reduce view. N is a number between 1 and 100. For instance, a _top_5 reducer will return the top 5 highest values for a group level.
#5498: Implement _first and _last reducers. These reducers return the first, and respectively, the last view row for a given group level. For example, _last can be used to retrieve the last timestamp update for each device_id if we had a key like [device_id, timestamp] if we query the view with group_level=1.
#5466: Conflict finder plugin. Enable the couch_scanner_plugin_conflict_finder plugin to find conflicting docs across all the databases. It can be configured to report individual revisions or aggregated statistics.

1.1.4.2. Performance

#5499: QuickJS rope based string implementation.
#5451: Optimize config system to use persistent terms.
#5437: Fix atts_since functionality for document GET requests. Avoids re-replicating attachment bodies on doc updates.
#5398: Save 1 write for each committing data to disk by using fdatasync while keeping the same level of storage reliability.

1.1.4.3. Features

#5526: Default upgrade_hash_on_auth to true. Downgrading to 3.4.1, 3.4.2, or 3.4.3 is safe. Those versions know how to verify these new password hashes.
#5517: Enable xxHash file checksums by default. Downgrading to 3.4.x versions should be safe. Those versions know how to read and verify xxHash checksums.
#5527: Update Fauxton and xxHash dependencies.
#5525: Array opcode optimization for QuickJS.
#5518: More precise error location reporting for QuickJS.
#5507, #5510, #5509: Erlang 28 compatibiliity.
#5516: Detailed node membership info for Prometheus.
#5489: Allow TLS client certs for Nouveau requests.
#5452: BigInt support for QuickJS.
#5439: Nouveau: upgrade dropwizard to 4.0.12.
#5429: Add simple+pbkdf2 migration password scheme.
#5424: Scanner: reduce log noise, fix QuickJS plugin mocks, gracefully handle broken search indexes.
#5421: Nouveau: upgrade Lucene to 9.12.1.
#5414: Remove unused multi_workers option from couch_work_queue.
#5402: Remove unused, undocumented and detrimental idle check timeout feature.
#5395: Remove unused, undocumented and unreliabele pread_limit feature from couch_file.
#5385: Clean up fabric_doc_update by introducing an #acc record.
#5372: Upgrade to Elixir 1.17.
#5351: Clouseau: show version in /_version endpoint.
#5338: Scanner: add Nouveau and Clouseau design doc validation.
#5335: Nouveau: support reading older Lucene 9x indexes.
#5327, #5329, #5419: Allow switching JavaScript engines at runtime.
#5326, #5328: Allow clients to specify HTTP request ID, including UUIDs.
#5321, #5366, #5413: Add support for SpiderMonkey versions 102, 115 and 128.
#5317: Add quickjs to the list of welcome features.

1.1.4.4. Bugfixes

#5515: Make sure query_limit config takes effect. Raise default limit from 2^28 to 2^59. Allow infinity as a more ergonomic config value.
#5522: Reopen indexes closed by Lucene in Nouveau.
#5508: Fix array from() and at() for QuickJS.
#5502: Buffer overflow and segfault fix in QuickJS.
#5469, #5471: Retry closed connections in Nouveau.
#5463: Fix badarith in Nouveau index query.
#5447: Fix arithmetic mean in _prometheus.
#5440: Fix _purged_infos when exceeding purged_infos_limit.
#5431: Restore the ability to return Error objects from map().
#5417: Clouseau: add a version check to connected() function to reliably detect if a Clouseau node is ready to be used.
#5416: Ensure we always map the documents in order in couch_mrview_updater. While views still built correctly, this behaviour simplifies debugging.
#5373: Fix checksumming in couch_file, consolidate similar functions and bring test coverage from 66% to 90%.
#5367: Scanner: be more resilient in the face of non-deterministic functions.
#5345: Scanner: be more resilient in the face of incomplete sample data.
#5344: Scanner: allow empty doc fields.
#5341: Improve Mango test reliability.
#5337: Prevent a broken mem3 app from permanently failing replication.
#5334: Fix QuickJS scanner function_clause error.
#5332: Skip deleted documents in the scanner.
#5331: Skip validation for design docs in the scanner.
#5330: Prevent inserting illegal design docs via Mango.

1.1.4.5. Docs

#5433: Mango: document Nouveau index type.
#5432: Add conceptual docs for Mango.
#5428: Fix wrong link in example in CONTRIBUTING.md.
#5400: Clarify RHEL9 installation caveats.
#5380, #5404: Fix various typos.
#5338: Clouseau: document version in /_version endpoint.
#5340, #5412: Nouveau: document search cleanup API.
#5316, #5325, #5426, #5442, #5445: Document various JavaScript engine incompatibilities, including SpiderMonkey 1.8.5 vs. newer SpiderMonkey and SpiderMonkey vs. QuickJS.
#5320, #5374: Improve auto-lockout feature documentation.
#5323: Nouveau: improve install instructions.

1.1.4.5.1. Tests

#5492: Enable Clouseau testing for FreeBSD
#5490: Enable Clouseau testing for MacOS
#5397: Fix negative-steps error in Elixir tests.

1.1.4.6. Builds

#5360: Use brew --prefix to find ICU paths on macOS.

1.1.4.7. Other

There’s always IOPS in the banana stand.