==> Building on electivire ==> Checking for remote environment... ==> Syncing package to remote host... sending incremental file list ./ PKGBUILD 700 9% 0.00kB/s 0:00:00 7,307 100% 6.30MB/s 0:00:00 (xfr#1, to-chk=25/27) allow-using-system-gradle.patch 700 48% 683.59kB/s 0:00:00 1,429 100% 1.36MB/s 0:00:00 (xfr#2, to-chk=24/27) skip-checks-for-git-repo.patch 700 32% 683.59kB/s 0:00:00 2,131 100% 2.03MB/s 0:00:00 (xfr#3, to-chk=23/27) skip-failing-test.patch 700 66% 683.59kB/s 0:00:00 1,047 100% 1022.46kB/s 0:00:00 (xfr#4, to-chk=22/27) solr-9.2.1-1.log 354 100% 345.70kB/s 0:00:00 354 100% 345.70kB/s 0:00:00 (xfr#5, to-chk=21/27) solr.service 700 72% 683.59kB/s 0:00:00 962 100% 939.45kB/s 0:00:00 (xfr#6, to-chk=20/27) solr.sysusers 32 100% 31.25kB/s 0:00:00 32 100% 31.25kB/s 0:00:00 (xfr#7, to-chk=19/27) solr.tmpfiles 161 100% 157.23kB/s 0:00:00 161 100% 157.23kB/s 0:00:00 (xfr#8, to-chk=18/27) support-reproducible-builds.patch 700 41% 683.59kB/s 0:00:00 1,668 100% 1.59MB/s 0:00:00 (xfr#9, to-chk=17/27) keys/ keys/pgp/ keys/pgp/2085660D9C1FCCACC4A479A3BF160FF14992A24C.asc 700 22% 683.59kB/s 0:00:00 3,078 100% 2.94MB/s 0:00:00 (xfr#10, to-chk=14/27) keys/pgp/3558857D1F5754B78C7F8B5A71A45A3D0D8D0B93.asc 700 41% 683.59kB/s 0:00:00 1,672 100% 1.59MB/s 0:00:00 (xfr#11, to-chk=13/27) keys/pgp/38DA0C3CE8181703A08E4D57377C3BA26AD29C0A.asc 700 22% 683.59kB/s 0:00:00 3,139 100% 2.99MB/s 0:00:00 (xfr#12, to-chk=12/27) keys/pgp/50E3EE1C91C7E0CB4DFB007B369424FC98F3F6EC.asc 700 22% 683.59kB/s 0:00:00 3,171 100% 3.02MB/s 0:00:00 (xfr#13, to-chk=11/27) keys/pgp/7D8D90F8F64F23077AC87CF7CB77CB79928BB0EC.asc 700 28% 683.59kB/s 0:00:00 2,448 100% 2.33MB/s 0:00:00 (xfr#14, to-chk=10/27) keys/pgp/81D3EB0408B4E1EB10AF443BA4F4C886B29BC2F4.asc 700 22% 683.59kB/s 0:00:00 3,163 100% 3.02MB/s 0:00:00 (xfr#15, to-chk=9/27) keys/pgp/86EDB9C33B8517228E88A8F93E48C0C6EF362B9E.asc 700 22% 341.80kB/s 0:00:00 3,090 100% 1.47MB/s 0:00:00 (xfr#16, to-chk=8/27) keys/pgp/902CC51935C140BF820230961FD5295281436075.asc 700 22% 341.80kB/s 0:00:00 3,167 100% 1.51MB/s 0:00:00 (xfr#17, to-chk=7/27) keys/pgp/9722F25F650057E26C803B60A6D064D833B3A969.asc 700 22% 341.80kB/s 0:00:00 3,159 100% 1.51MB/s 0:00:00 (xfr#18, to-chk=6/27) keys/pgp/C3E7CBD9B9FE2B419EB14B89612B39A5BC981763.asc 700 9% 341.80kB/s 0:00:00 7,770 100% 3.71MB/s 0:00:00 (xfr#19, to-chk=5/27) keys/pgp/CFCE5FBB920C3C745CEEE084C38FF5EC3FCFDB3E.asc 700 22% 341.80kB/s 0:00:00 3,090 100% 1.47MB/s 0:00:00 (xfr#20, to-chk=4/27) keys/pgp/E58A6F4D5B2B48AC66D5E53BD4F181881A42F9E6.asc 700 22% 227.86kB/s 0:00:00 3,155 100% 1.00MB/s 0:00:00 (xfr#21, to-chk=3/27) keys/pgp/E6E21FFCDCEA14C95910EA65051A0FAF76BC6507.asc 700 22% 227.86kB/s 0:00:00 3,094 100% 1007.16kB/s 0:00:00 (xfr#22, to-chk=2/27) keys/pgp/FBC25D7E1712025294FE66590A6AA179B9BBF45E.asc 700 22% 227.86kB/s 0:00:00 3,163 100% 1.01MB/s 0:00:00 (xfr#23, to-chk=1/27) keys/pgp/FDB3D11D716BB32ACF8C93AB919B21537AA80271.asc 700 22% 227.86kB/s 0:00:00 3,167 100% 1.01MB/s 0:00:00 (xfr#24, to-chk=0/27) sent 2,410 bytes received 1,133 bytes 2,362.00 bytes/sec total size is 64,460 speedup is 18.19 ==> Ensuring required PGP keys are present...  -> Checking for 2085660D9C1FCCACC4A479A3BF160FF14992A24C...  -> Checking for E58A6F4D5B2B48AC66D5E53BD4F181881A42F9E6...  -> Checking for 81D3EB0408B4E1EB10AF443BA4F4C886B29BC2F4...  -> Checking for 86EDB9C33B8517228E88A8F93E48C0C6EF362B9E...  -> Checking for 38DA0C3CE8181703A08E4D57377C3BA26AD29C0A...  -> Checking for 50E3EE1C91C7E0CB4DFB007B369424FC98F3F6EC...  -> Checking for 902CC51935C140BF820230961FD5295281436075...  -> Checking for 7D8D90F8F64F23077AC87CF7CB77CB79928BB0EC...  -> Checking for CFCE5FBB920C3C745CEEE084C38FF5EC3FCFDB3E...  -> Checking for FBC25D7E1712025294FE66590A6AA179B9BBF45E...  -> Checking for 9722F25F650057E26C803B60A6D064D833B3A969...  -> Checking for E6E21FFCDCEA14C95910EA65051A0FAF76BC6507...  -> Checking for 3558857D1F5754B78C7F8B5A71A45A3D0D8D0B93...  -> Checking for C3E7CBD9B9FE2B419EB14B89612B39A5BC981763...  -> Checking for FDB3D11D716BB32ACF8C93AB919B21537AA80271... ==> Running extra-riscv64-build -- -d /home/felix/packages/riscv64-pkg-cache:/var/cache/pacman/pkg -l root23 on remote host... [?25l:: Synchronizing package databases... core downloading... extra downloading... :: Starting full system upgrade... there is nothing to do [?25h==> Building in chroot for [extra] (riscv64)... ==> Synchronizing chroot copy [/var/lib/archbuild/extra-riscv64/root] -> [root23]...done ==> Making package: solr 9.2.1-1 (Sat Sep 2 22:07:55 2023) ==> Retrieving sources...  -> Found solr-9.2.1-src.tgz  -> Found solr-9.2.1-src.tgz.asc  -> Found solr.service  -> Found solr.sysusers  -> Found solr.tmpfiles  -> Found support-reproducible-builds.patch  -> Found skip-failing-test.patch  -> Found skip-checks-for-git-repo.patch  -> Found allow-using-system-gradle.patch ==> Validating source files with sha512sums... solr-9.2.1-src.tgz ... Passed solr-9.2.1-src.tgz.asc ... Skipped solr.service ... Passed solr.sysusers ... Passed solr.tmpfiles ... Passed support-reproducible-builds.patch ... Passed skip-failing-test.patch ... Passed skip-checks-for-git-repo.patch ... Passed allow-using-system-gradle.patch ... Passed ==> Validating source files with b2sums... solr-9.2.1-src.tgz ... Passed solr-9.2.1-src.tgz.asc ... Skipped solr.service ... Passed solr.sysusers ... Passed solr.tmpfiles ... Passed support-reproducible-builds.patch ... Passed skip-failing-test.patch ... Passed skip-checks-for-git-repo.patch ... Passed allow-using-system-gradle.patch ... Passed ==> Verifying source file signatures with gpg... solr-9.2.1-src.tgz ... Passed ==> Making package: solr 9.2.1-1 (Sat Sep 2 22:08:06 2023) ==> Checking runtime dependencies... ==> Installing missing dependencies... [?25lresolving dependencies... looking for conflicting packages... warning: dependency cycle detected: warning: harfbuzz will be installed before its freetype2 dependency Package (14) New Version Net Change extra/freetype2 2.13.1-1 1.51 MiB extra/giflib 5.2.1-2 0.22 MiB extra/graphite 1:1.3.14-3 0.17 MiB extra/harfbuzz 8.1.1-1 3.57 MiB extra/java-runtime-common 3-5 0.01 MiB extra/jre11-openjdk-headless 11.0.20.u8-2 138.33 MiB extra/lcms2 2.15-1 0.58 MiB extra/libjpeg-turbo 3.0.0-1 1.79 MiB extra/libnet 1:1.1.6-1 0.28 MiB extra/libpng 1.6.40-2 0.51 MiB extra/libtiff 4.5.1-1 5.72 MiB core/nspr 4.35-1 0.67 MiB core/nss 3.92-1 4.22 MiB extra/jre11-openjdk 11.0.20.u8-2 0.49 MiB Total Installed Size: 158.07 MiB :: Proceed with installation? [Y/n] checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing java-runtime-common... For the complete set of Java binaries to be available in your PATH, you need to re-login or source /etc/profile.d/jre.sh Please note that this package does not support forcing JAVA_HOME as former package java-common did installing nspr... installing nss... installing libjpeg-turbo... Optional dependencies for libjpeg-turbo java-runtime>11: for TurboJPEG Java wrapper installing libtiff... Optional dependencies for libtiff freeglut: for using tiffgt installing lcms2... installing libnet... installing libpng... installing graphite... Optional dependencies for graphite graphite-docs: Documentation installing harfbuzz... Optional dependencies for harfbuzz harfbuzz-utils: utilities installing freetype2... installing jre11-openjdk-headless... Optional dependencies for jre11-openjdk-headless java-rhino: for some JavaScript support installing giflib... installing jre11-openjdk... when you use a non-reparenting window manager, set _JAVA_AWT_WM_NONREPARENTING=1 in /etc/profile.d/jre.sh Optional dependencies for jre11-openjdk alsa-lib: for basic sound support gtk2: for the Gtk+ 2 look and feel - desktop usage gtk3: for the Gtk+ 3 look and feel - desktop usage [?25h==> Checking buildtime dependencies... ==> Installing missing dependencies... [?25lresolving dependencies... looking for conflicting packages... Package (9) New Version Net Change Download Size extra/hicolor-icon-theme 0.17-3 0.05 MiB extra/java-environment-common 3-5 0.00 MiB extra/perl-error 0.17029-5 0.04 MiB extra/perl-mailtools 2.21-7 0.10 MiB extra/perl-timedate 2.33-5 0.08 MiB extra/git 2.42.0-1 24.79 MiB extra/gradle 8.3-1 138.08 MiB 121.81 MiB extra/groovy 4.0.12-1 23.99 MiB extra/jdk11-openjdk 11.0.20.u8-2 247.04 MiB Total Download Size: 121.81 MiB Total Installed Size: 434.17 MiB :: Proceed with installation? [Y/n] :: Retrieving packages... gradle-8.3-1-any downloading... checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing java-environment-common... installing hicolor-icon-theme... installing jdk11-openjdk... installing gradle... Optional dependencies for gradle gradle-doc: gradle documentation gradle-src: gradle sources installing perl-error... installing perl-timedate... installing perl-mailtools... installing git... Optional dependencies for git tk: gitk and git gui openssh: ssh transport and crypto perl-libwww: git svn perl-term-readkey: git svn and interactive.singlekey setting perl-io-socket-ssl: git send-email TLS support perl-authen-sasl: git send-email TLS support perl-mediawiki-api: git mediawiki support perl-datetime-format-iso8601: git mediawiki support perl-lwp-protocol-https: git mediawiki https support perl-cgi: gitweb (web interface) support python: git svn & git p4 subversion: git svn org.freedesktop.secrets: keyring credential helper libsecret: libsecret credential helper [installed] installing groovy... Optional dependencies for groovy groovy-docs: html and pdf documentation for Groovy :: Running post-transaction hooks... (1/1) Warn about old perl modules [?25h==> Retrieving sources...  -> Found solr-9.2.1-src.tgz  -> Found solr-9.2.1-src.tgz.asc  -> Found solr.service  -> Found solr.sysusers  -> Found solr.tmpfiles  -> Found support-reproducible-builds.patch  -> Found skip-failing-test.patch  -> Found skip-checks-for-git-repo.patch  -> Found allow-using-system-gradle.patch ==> WARNING: Skipping all source file integrity checks. ==> Extracting sources...  -> Extracting solr-9.2.1-src.tgz with bsdtar ==> Starting prepare()... patching file gradle/globals.gradle patching file gradle/java/jar-manifest.gradle patching file solr/modules/extraction/src/test/org/apache/solr/handler/extraction/TestXLSXResponseWriter.java patching file solr/distribution/source-release.gradle patching file gradle/validation/precommit.gradle patching file solr/distribution/build.gradle ==> Starting build()... Downloading gradle-wrapper.jar from https://raw.githubusercontent.com/gradle/gradle/v7.6.0/gradle/wrapper/gradle-wrapper.jar Generating gradle.properties Downloading https://services.gradle.org/distributions/gradle-7.6-bin.zip ...........10%............20%...........30%............40%............50%...........60%............70%............80%...........90%............100% Welcome to Gradle 7.6! Here are the highlights of this release: - Added support for Java 19. - Introduced `--rerun` flag for individual task rerun. - Improved dependency block for test suites to be strongly typed. - Added a pluggable system for Java toolchains provisioning. For more details see https://docs.gradle.org/7.6/release-notes.html Starting a Gradle Daemon (subsequent builds will be faster) > Task :buildSrc:compileJava > Task :buildSrc:compileGroovy NO-SOURCE > Task :buildSrc:processResources NO-SOURCE > Task :buildSrc:classes > Task :buildSrc:jar > Task :buildSrc:assemble > Task :buildSrc:compileTestJava NO-SOURCE > Task :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test NO-SOURCE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build > Task :solr:modules:clustering:processResources NO-SOURCE > Task :mavenLocalClean UP-TO-DATE > Task :solr:solrj-zookeeper:processResources NO-SOURCE > Task :solr:modules:analysis-extras:processResources NO-SOURCE > Task :solr:modules:analytics:processResources NO-SOURCE > Task :solr:solrj:processResources NO-SOURCE > Task :solr:prometheus-exporter:processResources NO-SOURCE > Task :solr:modules:gcs-repository:processResources NO-SOURCE > Task :solr:core:processResources > Task :solr:solrj-streaming:processResources > Task :solr:modules:extraction:processResources > Task :solr:test-framework:processResources > Task :solr:prometheus-exporter:generatePomFileForJarsPublication > Task :solr:solrj:generatePomFileForJarsPublication > Task :solr:modules:analytics:generatePomFileForJarsPublication > Task :solr:modules:gcs-repository:generatePomFileForJarsPublication > Task :solr:test-framework:generatePomFileForJarsPublication > Task :solr:modules:analysis-extras:generatePomFileForJarsPublication > Task :solr:modules:clustering:generatePomFileForJarsPublication > Task :solr:solrj-zookeeper:generatePomFileForJarsPublication > Task :solr:solrj-streaming:generatePomFileForJarsPublication > Task :solr:modules:extraction:generatePomFileForJarsPublication > Task :solr:core:generatePomFileForJarsPublication > Task :gitStatus > Task :solr:modules:scripting:processResources NO-SOURCE > Task :solr:modules:ltr:processResources NO-SOURCE > Task :solr:modules:jaegertracer-configurator:processResources NO-SOURCE > Task :solr:modules:s3-repository:processResources NO-SOURCE > Task :solr:modules:hdfs:processResources NO-SOURCE > Task :solr:modules:opentelemetry:processResources NO-SOURCE > Task :solr:modules:jwt-auth:processResources NO-SOURCE > Task :errorProneSkipped WARNING: errorprone disabled (skipped on builds not running inside CI environments, pass -Pvalidation.errorprone=true to enable) > Task :solr:modules:hadoop-auth:processResources NO-SOURCE > Task :solr:modules:langid:processResources > Task :solr:docker:createBodySnippetDockerfile > Task :solr:modules:sql:processResources > Task :solr:docker:createDockerfileLocal > Task :solr:modules:ltr:generatePomFileForJarsPublication > Task :solr:modules:jaegertracer-configurator:generatePomFileForJarsPublication > Task :solr:modules:scripting:generatePomFileForJarsPublication > Task :solr:modules:langid:generatePomFileForJarsPublication > Task :solr:modules:jwt-auth:generatePomFileForJarsPublication > Task :solr:webapp:processResources NO-SOURCE > Task :solr:modules:hdfs:generatePomFileForJarsPublication > Task :solr:modules:hadoop-auth:generatePomFileForJarsPublication > Task :solr:modules:s3-repository:generatePomFileForJarsPublication > Task :solr:modules:sql:generatePomFileForJarsPublication > Task :solr:docker:assemblePackaging > Task :solr:distribution:prepareGitRev > Task :solr:modules:opentelemetry:generatePomFileForJarsPublication > Task :solr-missing-doclet:compileJava > Task :solr-missing-doclet:processResources NO-SOURCE > Task :solr-missing-doclet:classes > Task :solr-missing-doclet:jar > Task :solr:solrj:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:documentation:changesToHtml > Task :solr:documentation:copyDocumentationAssets > Task :solr:prometheus-exporter:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:documentation:markdownToHtml > Task :solr:documentation:copyChangesToHtmlForMiniSite > Task :solr:documentation:copyMiniDocumentationAssets > Task :solr:documentation:createMiniDocumentationIndex > Task :solr:documentation:documentationMinimal > Task :solr:solrj-zookeeper:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:solrj-zookeeper:classes > Task :solr:solrj-zookeeper:jar > Task :solr:solrj-zookeeper:generateMetadataFileForJarsPublication SKIPPED > Task :solr:solrj-zookeeper:sourcesJar > Task :solr:prometheus-exporter:classes > Task :solr:prometheus-exporter:jar > Task :solr:prometheus-exporter:generateMetadataFileForJarsPublication SKIPPED > Task :solr:prometheus-exporter:sourcesJar > Task :solr:solrj-streaming:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:solrj-streaming:classes > Task :solr:solrj-streaming:jar > Task :solr:solrj-streaming:generateMetadataFileForJarsPublication SKIPPED > Task :solr:solrj-streaming:sourcesJar > Task :solr:solrj:renderJavadoc > Task :solr:prometheus-exporter:assemblePackaging > Task :solr:solrj:classes > Task :solr:solrj:jar > Task :solr:solrj:generateMetadataFileForJarsPublication SKIPPED > Task :solr:solrj:javadoc SKIPPED > Task :solr:solrj:javadocJar > Task :solr:solrj:sourcesJar > Task :solr:solrj:publishJarsPublicationToBuildRepository > Task :solr:core:compileJava > Task :solr:prometheus-exporter:renderJavadoc > Task :solr:prometheus-exporter:javadoc SKIPPED > Task :solr:prometheus-exporter:javadocJar > Task :solr:prometheus-exporter:publishJarsPublicationToBuildRepository > Task :solr:solrj-zookeeper:renderJavadoc > Task :solr:solrj-zookeeper:javadoc SKIPPED > Task :solr:solrj-zookeeper:javadocJar > Task :solr:solrj-zookeeper:publishJarsPublicationToBuildRepository > Task :solr:solrj-streaming:renderJavadoc > Task :solr:solrj-streaming:javadoc SKIPPED > Task :solr:solrj-streaming:javadocJar > Task :solr:solrj-streaming:publishJarsPublicationToBuildRepository > Task :solr:core:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:core:classes > Task :solr:core:jar > Task :solr:core:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:jwt-auth:compileJava Note: /build/solr/src/solr-9.2.1/solr/modules/jwt-auth/src/java/org/apache/solr/security/jwt/JWTAuthPlugin.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:modules:jaegertracer-configurator:compileJava > Task :solr:modules:jaegertracer-configurator:classes > Task :solr:modules:analysis-extras:compileJava Note: /build/solr/src/solr-9.2.1/solr/modules/analysis-extras/src/java/org/apache/solr/schema/ICUCollationField.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:modules:jaegertracer-configurator:jar > Task :solr:modules:jaegertracer-configurator:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:langid:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:modules:jaegertracer-configurator:sourcesJar > Task :solr:modules:clustering:compileJava > Task :solr:modules:clustering:classes > Task :solr:modules:clustering:jar > Task :solr:modules:clustering:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:clustering:sourcesJar > Task :solr:modules:jwt-auth:classes > Task :solr:modules:langid:classes > Task :solr:modules:jwt-auth:jar > Task :solr:modules:jwt-auth:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:jwt-auth:sourcesJar > Task :solr:modules:langid:jar > Task :solr:modules:langid:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:langid:sourcesJar > Task :solr:modules:hadoop-auth:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:modules:analysis-extras:classes > Task :solr:modules:analysis-extras:jar > Task :solr:modules:analysis-extras:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:analysis-extras:sourcesJar > Task :solr:modules:scripting:compileJava Note: /build/solr/src/solr-9.2.1/solr/modules/scripting/src/java/org/apache/solr/scripting/xslt/XSLTUpdateRequestHandler.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:modules:scripting:classes > Task :solr:modules:scripting:jar > Task :solr:modules:scripting:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:scripting:sourcesJar > Task :solr:webapp:compileJava NO-SOURCE > Task :solr:webapp:classes UP-TO-DATE > Task :solr:modules:opentelemetry:compileJava > Task :solr:modules:opentelemetry:classes > Task :solr:modules:opentelemetry:jar > Task :solr:modules:opentelemetry:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:opentelemetry:sourcesJar > Task :solr:modules:ltr:compileJava > Task :solr:modules:ltr:classes > Task :solr:modules:ltr:jar > Task :solr:modules:ltr:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:ltr:sourcesJar > Task :solr:modules:clustering:assemblePackaging > Task :solr:modules:hadoop-auth:classes > Task :solr:modules:analysis-extras:assemblePackaging > Task :solr:modules:gcs-repository:compileJava Note: /build/solr/src/solr-9.2.1/solr/modules/gcs-repository/src/java/org/apache/solr/gcs/GCSConfigParser.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:modules:hadoop-auth:jar > Task :solr:modules:hadoop-auth:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:hadoop-auth:sourcesJar > Task :solr:modules:jaegertracer-configurator:assemblePackaging > Task :solr:modules:jwt-auth:assemblePackaging > Task :solr:modules:ltr:assemblePackaging > Task :solr:modules:langid:assemblePackaging > Task :solr:modules:scripting:assemblePackaging > Task :solr:modules:hdfs:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:test-framework:compileJava > Task :solr:modules:analytics:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:modules:extraction:compileJava > Task :solr:modules:analytics:classes > Task :solr:modules:analytics:jar > Task :solr:modules:analytics:generateMetadataFileForJarsPublication SKIPPED > Task :solr:core:renderJavadoc > Task :solr:modules:analytics:sourcesJar > Task :solr:modules:sql:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:webapp:war > Task :solr:modules:analytics:assemblePackaging > Task :solr:test-framework:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:server:assemblePackaging > Task :solr:modules:extraction:compileJava Note: /build/solr/src/solr-9.2.1/solr/modules/extraction/src/java/org/apache/solr/handler/extraction/ExtractingDocumentLoader.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:modules:opentelemetry:assemblePackaging > Task :solr:modules:hadoop-auth:assemblePackaging > Task :solr:modules:s3-repository:compileJava > Task :solr:modules:s3-repository:classes > Task :solr:modules:s3-repository:jar > Task :solr:modules:s3-repository:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:s3-repository:sourcesJar > Task :solr:modules:s3-repository:assemblePackaging > Task :solr:test-framework:classes > Task :solr:test-framework:jar > Task :solr:test-framework:generateMetadataFileForJarsPublication SKIPPED > Task :solr:test-framework:sourcesJar > Task :solr:modules:gcs-repository:classes > Task :solr:modules:gcs-repository:jar > Task :solr:modules:gcs-repository:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:gcs-repository:sourcesJar > Task :solr:modules:sql:classes > Task :solr:modules:sql:jar > Task :solr:modules:sql:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:sql:sourcesJar > Task :solr:modules:gcs-repository:assemblePackaging > Task :solr:modules:sql:assemblePackaging > Task :solr:modules:extraction:classes > Task :solr:modules:extraction:jar > Task :solr:modules:extraction:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:extraction:sourcesJar > Task :solr:modules:extraction:assemblePackaging > Task :solr:modules:hdfs:classes > Task :solr:modules:hdfs:jar > Task :solr:modules:hdfs:generateMetadataFileForJarsPublication SKIPPED > Task :solr:modules:hdfs:sourcesJar > Task :solr:modules:hdfs:assemblePackaging > Task :solr:core:javadoc SKIPPED > Task :solr:core:javadocJar > Task :solr:core:sourcesJar > Task :solr:core:publishJarsPublicationToBuildRepository > Task :solr:modules:jaegertracer-configurator:renderJavadoc > Task :solr:modules:jaegertracer-configurator:javadoc SKIPPED > Task :solr:modules:jaegertracer-configurator:javadocJar > Task :solr:modules:jaegertracer-configurator:publishJarsPublicationToBuildRepository > Task :solr:modules:clustering:renderJavadoc > Task :solr:modules:clustering:javadoc SKIPPED > Task :solr:modules:clustering:javadocJar > Task :solr:modules:clustering:publishJarsPublicationToBuildRepository > Task :solr:modules:analysis-extras:renderJavadoc > Task :solr:modules:analysis-extras:javadoc SKIPPED > Task :solr:modules:analysis-extras:javadocJar > Task :solr:modules:analysis-extras:publishJarsPublicationToBuildRepository > Task :solr:modules:langid:renderJavadoc > Task :solr:modules:langid:javadoc SKIPPED > Task :solr:modules:langid:javadocJar > Task :solr:modules:langid:publishJarsPublicationToBuildRepository > Task :solr:modules:gcs-repository:renderJavadoc > Task :solr:modules:hadoop-auth:renderJavadoc > Task :solr:modules:hadoop-auth:javadoc SKIPPED > Task :solr:modules:gcs-repository:javadoc SKIPPED > Task :solr:modules:jwt-auth:renderJavadoc > Task :solr:modules:jwt-auth:javadoc SKIPPED > Task :solr:modules:gcs-repository:javadocJar > Task :solr:modules:hadoop-auth:javadocJar > Task :solr:modules:gcs-repository:publishJarsPublicationToBuildRepository > Task :solr:modules:hadoop-auth:publishJarsPublicationToBuildRepository > Task :solr:modules:jwt-auth:javadocJar > Task :solr:core:assemblePostJar > Task :solr:modules:jwt-auth:publishJarsPublicationToBuildRepository > Task :solr:example:assemblePackaging > Task :solr:modules:extraction:renderJavadoc > Task :solr:modules:extraction:javadoc SKIPPED > Task :solr:modules:extraction:javadocJar > Task :solr:modules:extraction:publishJarsPublicationToBuildRepository > Task :solr:modules:hdfs:renderJavadoc > Task :solr:modules:hdfs:javadoc SKIPPED > Task :solr:modules:hdfs:javadocJar > Task :solr:modules:hdfs:publishJarsPublicationToBuildRepository > Task :solr:modules:opentelemetry:renderJavadoc > Task :solr:modules:opentelemetry:javadoc SKIPPED > Task :solr:modules:opentelemetry:javadocJar > Task :solr:modules:opentelemetry:publishJarsPublicationToBuildRepository > Task :solr:modules:scripting:renderJavadoc > Task :solr:modules:scripting:javadoc SKIPPED > Task :solr:modules:scripting:javadocJar > Task :solr:modules:scripting:publishJarsPublicationToBuildRepository > Task :solr:modules:s3-repository:renderJavadoc > Task :solr:modules:s3-repository:javadoc SKIPPED > Task :solr:modules:s3-repository:javadocJar > Task :solr:modules:s3-repository:publishJarsPublicationToBuildRepository > Task :solr:modules:ltr:renderJavadoc > Task :solr:modules:ltr:javadoc SKIPPED > Task :solr:test-framework:renderJavadoc > Task :solr:modules:analytics:renderJavadoc > Task :solr:modules:ltr:javadocJar > Task :solr:modules:ltr:publishJarsPublicationToBuildRepository > Task :solr:modules:sql:renderJavadoc > Task :solr:modules:sql:javadoc SKIPPED > Task :solr:modules:sql:javadocJar > Task :solr:modules:sql:publishJarsPublicationToBuildRepository > Task :solr:test-framework:javadoc SKIPPED > Task :solr:test-framework:javadocJar > Task :solr:test-framework:publishJarsPublicationToBuildRepository > Task :solr:modules:analytics:javadoc SKIPPED > Task :solr:modules:analytics:javadocJar > Task :solr:modules:analytics:publishJarsPublicationToBuildRepository > Task :mavenToLocalFolder Local maven artifacts (poms, jars) created at: /build/solr/src/solr-9.2.1/build/maven-local > Task :solr:packaging:distTar > Task :solr:distribution:computeChecksums > Task :solr:distribution:assembleRelease BUILD SUCCESSFUL in 9m 40s 194 actionable tasks: 193 executed, 1 up-to-date /usr/bin/install: creating directory '/build/solr/src/solr-9.2.1/dist' ==> Starting check()... > Task :buildSrc:compileJava UP-TO-DATE > Task :buildSrc:compileGroovy NO-SOURCE > Task :buildSrc:processResources NO-SOURCE > Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE > Task :buildSrc:assemble UP-TO-DATE > Task :buildSrc:compileTestJava NO-SOURCE > Task :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test NO-SOURCE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build UP-TO-DATE > Task :solr:prometheus-exporter:processResources NO-SOURCE > Task :solr:solrj-zookeeper:processResources NO-SOURCE > Task :solr:solrj:processResources NO-SOURCE > Task :solr:modules:analysis-extras:processResources NO-SOURCE > Task :solr:server:processResources NO-SOURCE > Task :solr:solr-ref-guide:processResources NO-SOURCE > Task :randomizationInfo Running tests with randomization seed: tests.seed=DF064544E215B325 > Task :solr:test-framework:processResources UP-TO-DATE > Task :solr:core:processResources UP-TO-DATE > Task :solr:modules:sql:processResources UP-TO-DATE > Task :solr:solrj-streaming:processResources UP-TO-DATE > Task :errorProneSkipped WARNING: errorprone disabled (skipped on builds not running inside CI environments, pass -Pvalidation.errorprone=true to enable) > Task :solr:server:processTestResources NO-SOURCE > Task :solr:server:compileJava SKIPPED > Task :gitStatus > Task :solr:server:classes UP-TO-DATE > Task :solr:server:compileTestJava NO-SOURCE > Task :solr:server:testClasses UP-TO-DATE > Task :solr:modules:analysis-extras:processTestResources > Task :solr:modules:clustering:processResources NO-SOURCE > Task :solr:modules:analytics:processResources NO-SOURCE > Task :solr:server:test NO-SOURCE > Task :solr:server:wipeTaskTemp > Task :solr:solr-ref-guide:copySolrjTestResources > Task :solr:modules:extraction:processResources UP-TO-DATE > Task :solr:solr-ref-guide:compileJava NO-SOURCE > Task :solr:solr-ref-guide:classes UP-TO-DATE > Task :solr:solrj:processTestResources > Task :solr:modules:extraction:processTestResources > Task :solr:modules:gcs-repository:processResources NO-SOURCE > Task :solr:modules:gcs-repository:processTestResources > Task :solr:webapp:processResources NO-SOURCE > Task :solr:webapp:processTestResources NO-SOURCE > Task :solr:solrj-zookeeper:processTestResources > Task :solr:solr-ref-guide:processTestResources NO-SOURCE > Task :solr:modules:hdfs:processResources NO-SOURCE > Task :solr:test-framework:processTestResources > Task :solr:modules:jaegertracer-configurator:processResources NO-SOURCE > Task :solr:solrj-streaming:processTestResources > Task :solr:modules:analytics:processTestResources > Task :solr:modules:opentelemetry:processResources NO-SOURCE > Task :solr:modules:langid:processResources UP-TO-DATE > Task :solr:prometheus-exporter:processTestResources > Task :solr:modules:ltr:processResources NO-SOURCE > Task :solr:modules:jaegertracer-configurator:processTestResources > Task :solr:modules:s3-repository:processResources NO-SOURCE > Task :solr:modules:hadoop-auth:processResources NO-SOURCE > Task :solr:modules:jwt-auth:processResources NO-SOURCE > Task :solr:benchmark:processResources > Task :solr:modules:scripting:processResources NO-SOURCE > Task :solr:modules:clustering:processTestResources > Task :solr:modules:opentelemetry:processTestResources > Task :solr:modules:s3-repository:processTestResources > Task :solr:modules:langid:processTestResources > Task :solr:benchmark:processTestResources > Task :solr:modules:scripting:processTestResources > Task :solr:modules:jwt-auth:processTestResources > Task :solr:solrj:compileJava UP-TO-DATE > Task :solr:solrj:classes UP-TO-DATE > Task :solr:modules:ltr:processTestResources > Task :solr:solrj-streaming:compileJava UP-TO-DATE > Task :solr:solrj-zookeeper:compileJava UP-TO-DATE > Task :solr:prometheus-exporter:compileJava UP-TO-DATE > Task :solr:solrj:jar UP-TO-DATE > Task :solr:solrj-streaming:classes UP-TO-DATE > Task :solr:prometheus-exporter:classes UP-TO-DATE > Task :solr:solrj-zookeeper:classes UP-TO-DATE > Task :solr:solrj-streaming:jar UP-TO-DATE > Task :solr:solrj-zookeeper:jar UP-TO-DATE > Task :solr:modules:hdfs:copySolrCoreTestResources > Task :solr:core:processTestResources > Task :solr:modules:hadoop-auth:copySolrCoreTestResources > Task :solr:modules:sql:copySolrCoreTestResources > Task :solr:modules:hdfs:processTestResources > Task :solr:modules:hadoop-auth:processTestResources > Task :solr:modules:sql:processTestResources > Task :solr:core:compileJava UP-TO-DATE > Task :solr:core:classes UP-TO-DATE > Task :solr:core:jar UP-TO-DATE > Task :solr:modules:clustering:compileJava UP-TO-DATE > Task :solr:modules:clustering:classes UP-TO-DATE > Task :solr:webapp:compileJava NO-SOURCE > Task :solr:webapp:classes UP-TO-DATE > Task :solr:webapp:compileTestJava NO-SOURCE > Task :solr:webapp:testClasses UP-TO-DATE > Task :solr:modules:hadoop-auth:compileJava UP-TO-DATE > Task :solr:modules:hdfs:compileJava UP-TO-DATE > Task :solr:modules:analysis-extras:compileJava UP-TO-DATE > Task :solr:modules:sql:compileJava UP-TO-DATE > Task :solr:modules:gcs-repository:compileJava UP-TO-DATE > Task :solr:modules:analytics:compileJava UP-TO-DATE > Task :solr:modules:jaegertracer-configurator:compileJava UP-TO-DATE > Task :solr:modules:hdfs:classes UP-TO-DATE > Task :solr:modules:analysis-extras:classes UP-TO-DATE > Task :solr:modules:hadoop-auth:classes UP-TO-DATE > Task :solr:modules:sql:classes UP-TO-DATE > Task :solr:webapp:test NO-SOURCE > Task :solr:modules:analytics:classes UP-TO-DATE > Task :solr:modules:gcs-repository:classes UP-TO-DATE > Task :solr:modules:jaegertracer-configurator:classes UP-TO-DATE > Task :solr:webapp:wipeTaskTemp > Task :solr:modules:jwt-auth:compileJava UP-TO-DATE > Task :solr:modules:langid:compileJava UP-TO-DATE > Task :solr:modules:scripting:compileJava UP-TO-DATE > Task :solr:modules:ltr:compileJava UP-TO-DATE > Task :solr:test-framework:compileJava UP-TO-DATE > Task :solr:modules:analysis-extras:jar UP-TO-DATE > Task :solr:modules:jwt-auth:classes UP-TO-DATE > Task :solr:modules:langid:classes UP-TO-DATE > Task :solr:test-framework:classes UP-TO-DATE > Task :solr:modules:sql:jar UP-TO-DATE > Task :solr:modules:opentelemetry:compileJava UP-TO-DATE > Task :solr:modules:s3-repository:compileJava UP-TO-DATE > Task :solr:modules:extraction:compileJava UP-TO-DATE > Task :solr:modules:extraction:classes UP-TO-DATE > Task :solr:test-framework:jar UP-TO-DATE > Task :solr:modules:extraction:compileTestJava Note: /build/solr/src/solr-9.2.1/solr/modules/extraction/src/test/org/apache/solr/handler/extraction/ExtractingRequestHandlerTest.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:solr-ref-guide:compileTestJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:solrj-zookeeper:compileTestJava > Task :solr:test-framework:compileTestJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:solrj-zookeeper:testClasses > Task :solr:test-framework:testClasses > Task :solr:modules:extraction:testClasses > Task :solr:solr-ref-guide:testClasses > Task :solr:prometheus-exporter:compileTestJava Note: /build/solr/src/solr-9.2.1/solr/prometheus-exporter/src/test/org/apache/solr/prometheus/exporter/MetricsQueryTemplateTest.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:modules:analysis-extras:compileTestJava Note: /build/solr/src/solr-9.2.1/solr/modules/analysis-extras/src/test/org/apache/solr/schema/TestICUCollationFieldUDVAS.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:prometheus-exporter:testClasses > Task :solr:modules:clustering:compileTestJava > Task :solr:modules:clustering:testClasses > Task :solr:solrj:compileTestJava > Task :solr:benchmark:compileJava > Task :solr:core:compileTestJava > Task :solr:solrj-streaming:compileTestJava > Task :solr:modules:analytics:compileTestJava > Task :solr:modules:analysis-extras:testClasses > Task :solr:modules:analytics:compileTestJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:benchmark:classes > Task :solr:modules:analytics:testClasses > Task :solr:benchmark:compileTestJava > Task :solr:solrj-streaming:compileTestJava Note: /build/solr/src/solr-9.2.1/solr/solrj-streaming/src/test/org/apache/solr/client/solrj/io/stream/StreamingTest.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:benchmark:testClasses > Task :solr:modules:gcs-repository:compileTestJava Note: /build/solr/src/solr-9.2.1/solr/modules/gcs-repository/src/test/org/apache/solr/gcs/ConcurrentDelegatingStorage.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:solrj-streaming:testClasses > Task :solr:solrj-zookeeper:test > Task :solr:modules:extraction:test > Task :solr:test-framework:test > Task :solr:solr-ref-guide:test > Task :solr:prometheus-exporter:test > Task :solr:modules:clustering:test > Task :solr:solrj:compileTestJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:solrj:testClasses > Task :solr:modules:analysis-extras:test > Task :solr:modules:analytics:test > Task :solr:benchmark:test > Task :solr:solrj-streaming:test > Task :solr:solrj:test > Task :solr:core:compileTestJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:core:testClasses > Task :solr:core:test > Task :solr:benchmark:test :solr:benchmark:test (SUCCESS): 11 test(s) > Task :solr:solr-ref-guide:test :solr:solr-ref-guide:test (SUCCESS): 32 test(s) > Task :solr:modules:clustering:test :solr:modules:clustering:test (SUCCESS): 15 test(s) > Task :solr:benchmark:wipeTaskTemp > Task :solr:modules:gcs-repository:testClasses > Task :solr:modules:gcs-repository:test > Task :solr:modules:extraction:test :solr:modules:extraction:test (SUCCESS): 20 test(s), 3 skipped > Task :solr:core:test org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest > deletePartiallyCreatedCollection FAILED org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33017/solr: Underlying core creation failed while creating collection: halfdeletedcollection at __randomizedtesting.SeedInfo.seed([DF064544E215B325:F58629A155CF93ED]:0) at app//org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:745) at app//org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259) at app//org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:240) at app//org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:386) at app//org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:352) at app//org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1184) at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:890) at app//org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:818) at app//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:234) at app//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:249) at app//org.apache.solr.cloud.api.collections.AbstractCollectionsAPIDistributedZkTestBase.deletePartiallyCreatedCollection(AbstractCollectionsAPIDistributedZkTestBase.java:166) at java.base@11.0.20/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base@11.0.20/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base@11.0.20/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base@11.0.20/java.lang.reflect.Method.invoke(Method.java:566) at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) at app//org.junit.rules.RunRules.evaluate(RunRules.java:20) at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44) at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) at app//org.junit.rules.RunRules.evaluate(RunRules.java:20) at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) at app//org.junit.rules.RunRules.evaluate(RunRules.java:20) at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) at app//org.junit.rules.RunRules.evaluate(RunRules.java:20) at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) at java.base@11.0.20/java.lang.Thread.run(Thread.java:829) org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest > test suite's output saved to /build/solr/src/solr-9.2.1/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.txt, copied below: 2> 306188 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/build/solr/src/solr-9.2.1/solr/server/solr/configsets/_default/conf' 2> 306190 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom 2> 306194 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-42 after mutting 0 log messages 2> 306194 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-43 for ERROR logs matching regex: ignore_exception 2> 306197 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.SolrTestCaseJ4 Created dataDir: /build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/data-dir-16-001 2> 306198 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false 2> 306201 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0) 2> 306206 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001 2> 306208 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER 2> 306209 INFO (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer client port: 0.0.0.0/0.0.0.0:0 2> 306209 INFO (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer Starting server 2> 306214 WARN (ZkTestServer Run Thread) [] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0. 2> 306309 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.ZkTestServer start zk server on port: 33795 2> 306316 WARN (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 306320 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2> 306332 INFO (zkConnectionManagerCallback-1297-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected 2> 306332 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2> 306333 WARN (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 306341 WARN (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 306345 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2> 306350 INFO (zkConnectionManagerCallback-1299-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected 2> 306350 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2> 306350 WARN (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 306351 WARN (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 306353 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2> 306359 INFO (zkConnectionManagerCallback-1301-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected 2> 306360 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2> 306360 WARN (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 306488 INFO (jetty-launcher-1302-thread-1) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0) 2> 306488 INFO (jetty-launcher-1302-thread-2) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0) 2> 306488 INFO (jetty-launcher-1302-thread-4) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0) 2> 306488 INFO (jetty-launcher-1302-thread-3) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0) 2> 306489 INFO (jetty-launcher-1302-thread-1) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ... 2> 306489 INFO (jetty-launcher-1302-thread-2) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ... 2> 306489 INFO (jetty-launcher-1302-thread-3) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ... 2> 306489 INFO (jetty-launcher-1302-thread-4) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ... 2> 306490 INFO (jetty-launcher-1302-thread-1) [] o.e.j.s.Server jetty-10.0.13; built: 2022-12-07T20:13:20.134Z; git: 1c2636ea05c0ca8de1ffd6ca7f3a98ac084c766d; jvm 11.0.20+8 2> 306490 INFO (jetty-launcher-1302-thread-2) [] o.e.j.s.Server jetty-10.0.13; built: 2022-12-07T20:13:20.134Z; git: 1c2636ea05c0ca8de1ffd6ca7f3a98ac084c766d; jvm 11.0.20+8 2> 306490 INFO (jetty-launcher-1302-thread-4) [] o.e.j.s.Server jetty-10.0.13; built: 2022-12-07T20:13:20.134Z; git: 1c2636ea05c0ca8de1ffd6ca7f3a98ac084c766d; jvm 11.0.20+8 2> 306490 INFO (jetty-launcher-1302-thread-3) [] o.e.j.s.Server jetty-10.0.13; built: 2022-12-07T20:13:20.134Z; git: 1c2636ea05c0ca8de1ffd6ca7f3a98ac084c766d; jvm 11.0.20+8 2> 306496 INFO (jetty-launcher-1302-thread-4) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0 2> 306497 INFO (jetty-launcher-1302-thread-2) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0 2> 306499 INFO (jetty-launcher-1302-thread-1) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0 2> 306499 INFO (jetty-launcher-1302-thread-3) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0 2> 306506 INFO (jetty-launcher-1302-thread-4) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@c8983c4{/solr,file:///build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-cwd/,AVAILABLE} 2> 306506 INFO (jetty-launcher-1302-thread-3) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6c11fe{/solr,file:///build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-cwd/,AVAILABLE} 2> 306506 INFO (jetty-launcher-1302-thread-2) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@780b670c{/solr,file:///build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-cwd/,AVAILABLE} 2> 306506 INFO (jetty-launcher-1302-thread-1) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4dca3202{/solr,file:///build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-cwd/,AVAILABLE} 2> 306509 INFO (jetty-launcher-1302-thread-3) [] o.e.j.s.AbstractConnector Started ServerConnector@3073408f{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:37221} 2> 306509 INFO (jetty-launcher-1302-thread-4) [] o.e.j.s.AbstractConnector Started ServerConnector@45e3b957{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:33017} 2> 306509 INFO (jetty-launcher-1302-thread-2) [] o.e.j.s.AbstractConnector Started ServerConnector@20ca1cbb{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:40837} 2> 306509 INFO (jetty-launcher-1302-thread-1) [] o.e.j.s.AbstractConnector Started ServerConnector@27706969{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:33871} 2> 306511 INFO (jetty-launcher-1302-thread-1) [] o.e.j.s.Server Started Server@57e1a769{STARTING}[10.0.13,sto=0] @306741ms 2> 306511 INFO (jetty-launcher-1302-thread-4) [] o.e.j.s.Server Started Server@35901618{STARTING}[10.0.13,sto=0] @306741ms 2> 306511 INFO (jetty-launcher-1302-thread-2) [] o.e.j.s.Server Started Server@50dcf77c{STARTING}[10.0.13,sto=0] @306741ms 2> 306511 INFO (jetty-launcher-1302-thread-3) [] o.e.j.s.Server Started Server@40f59aa7{STARTING}[10.0.13,sto=0] @306741ms 2> 306512 ERROR (jetty-launcher-1302-thread-2) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete. 2> 306512 ERROR (jetty-launcher-1302-thread-1) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete. 2> 306512 ERROR (jetty-launcher-1302-thread-3) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete. 2> 306512 INFO (jetty-launcher-1302-thread-1) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2> 306512 ERROR (jetty-launcher-1302-thread-4) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete. 2> 306512 INFO (jetty-launcher-1302-thread-2) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2> 306512 INFO (jetty-launcher-1302-thread-3) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2> 306512 INFO (jetty-launcher-1302-thread-4) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2> 306512 INFO (jetty-launcher-1302-thread-1) [] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.2.1 2> 306512 INFO (jetty-launcher-1302-thread-3) [] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.2.1 2> 306512 INFO (jetty-launcher-1302-thread-4) [] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.2.1 2> 306512 INFO (jetty-launcher-1302-thread-2) [] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.2.1 2> 306514 INFO (jetty-launcher-1302-thread-3) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port null 2> 306514 INFO (jetty-launcher-1302-thread-1) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port null 2> 306514 INFO (jetty-launcher-1302-thread-4) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port null 2> 306514 INFO (jetty-launcher-1302-thread-2) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port null 2> 306515 INFO (jetty-launcher-1302-thread-2) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /build/solr/src/solr-9.2.1/solr 2> 306515 INFO (jetty-launcher-1302-thread-1) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /build/solr/src/solr-9.2.1/solr 2> 306515 INFO (jetty-launcher-1302-thread-4) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /build/solr/src/solr-9.2.1/solr 2> 306515 INFO (jetty-launcher-1302-thread-3) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /build/solr/src/solr-9.2.1/solr 2> 306515 INFO (jetty-launcher-1302-thread-1) [] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2023-09-02T14:30:45.481675Z 2> 306515 INFO (jetty-launcher-1302-thread-4) [] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2023-09-02T14:30:45.481748Z 2> 306515 INFO (jetty-launcher-1302-thread-2) [] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2023-09-02T14:30:45.481633Z 2> 306515 INFO (jetty-launcher-1302-thread-3) [] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2023-09-02T14:30:45.481815Z 2> 306517 INFO (jetty-launcher-1302-thread-1) [] o.a.s.s.CoreContainerProvider Solr Home: /build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1 (source: servlet config: solr.solr.home) 2> 306518 INFO (jetty-launcher-1302-thread-3) [] o.a.s.s.CoreContainerProvider Solr Home: /build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node3 (source: servlet config: solr.solr.home) 2> 306518 INFO (jetty-launcher-1302-thread-2) [] o.a.s.s.CoreContainerProvider Solr Home: /build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node2 (source: servlet config: solr.solr.home) 2> 306518 INFO (jetty-launcher-1302-thread-4) [] o.a.s.s.CoreContainerProvider Solr Home: /build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4 (source: servlet config: solr.solr.home) 2> 306518 WARN (jetty-launcher-1302-thread-1) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 306518 WARN (jetty-launcher-1302-thread-3) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 306518 WARN (jetty-launcher-1302-thread-2) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 306518 WARN (jetty-launcher-1302-thread-4) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 306521 INFO (jetty-launcher-1302-thread-1) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2> 306524 INFO (jetty-launcher-1302-thread-2) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2> 306524 INFO (jetty-launcher-1302-thread-3) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2> 306524 INFO (jetty-launcher-1302-thread-4) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2> 306530 INFO (zkConnectionManagerCallback-1310-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected 2> 306530 INFO (jetty-launcher-1302-thread-4) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2> 306530 WARN (jetty-launcher-1302-thread-4) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 306530 INFO (zkConnectionManagerCallback-1304-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected 2> 306531 INFO (zkConnectionManagerCallback-1308-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected 2> 306531 INFO (jetty-launcher-1302-thread-1) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2> 306531 INFO (jetty-launcher-1302-thread-2) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2> 306531 WARN (jetty-launcher-1302-thread-1) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 306531 WARN (jetty-launcher-1302-thread-2) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 306532 INFO (zkConnectionManagerCallback-1306-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected 2> 306532 INFO (jetty-launcher-1302-thread-4) [] o.a.s.c.NodeConfig solr.xml found in ZooKeeper. Loading... 2> 306532 INFO (jetty-launcher-1302-thread-3) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2> 306532 WARN (jetty-launcher-1302-thread-3) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 306533 INFO (jetty-launcher-1302-thread-2) [] o.a.s.c.NodeConfig solr.xml found in ZooKeeper. Loading... 2> 306534 INFO (jetty-launcher-1302-thread-1) [] o.a.s.c.NodeConfig solr.xml found in ZooKeeper. Loading... 2> 306534 INFO (jetty-launcher-1302-thread-3) [] o.a.s.c.NodeConfig solr.xml found in ZooKeeper. Loading... 2> 307795 INFO (jetty-launcher-1302-thread-1) [] o.a.s.c.CoreContainer Allowing use of paths: [_ALL_] 2> 307824 INFO (jetty-launcher-1302-thread-1) [] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33795/solr 2> 307824 INFO (jetty-launcher-1302-thread-1) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2> 307829 INFO (jetty-launcher-1302-thread-1) [n:127.0.0.1:33871_solr] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2> 307836 INFO (zkConnectionManagerCallback-1329-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected 2> 307836 INFO (jetty-launcher-1302-thread-1) [n:127.0.0.1:33871_solr] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2> 307867 INFO (jetty-launcher-1302-thread-4) [] o.a.s.c.CoreContainer Allowing use of paths: [_ALL_] 2> 307875 WARN (jetty-launcher-1302-thread-1) [n:127.0.0.1:33871_solr] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2> 307882 INFO (jetty-launcher-1302-thread-1) [n:127.0.0.1:33871_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2> 307890 INFO (jetty-launcher-1302-thread-4) [] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33795/solr 2> 307890 INFO (jetty-launcher-1302-thread-1) [n:127.0.0.1:33871_solr] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33871_solr 2> 307890 INFO (jetty-launcher-1302-thread-4) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2> 307893 INFO (jetty-launcher-1302-thread-1) [n:127.0.0.1:33871_solr] o.a.s.c.Overseer Overseer (id=72085948270444551-127.0.0.1:33871_solr-n_0000000000) starting 2> 307930 INFO (jetty-launcher-1302-thread-1) [n:127.0.0.1:33871_solr] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33871_solr 2> 307933 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:33871_solr 2> 307937 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2> 307947 INFO (jetty-launcher-1302-thread-4) [n:127.0.0.1:33017_solr] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2> 307954 INFO (zkConnectionManagerCallback-1338-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected 2> 307954 INFO (jetty-launcher-1302-thread-4) [n:127.0.0.1:33017_solr] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2> 307959 WARN (jetty-launcher-1302-thread-1) [n:127.0.0.1:33871_solr] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2> 307969 WARN (jetty-launcher-1302-thread-4) [n:127.0.0.1:33017_solr] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2> 307982 INFO (jetty-launcher-1302-thread-4) [n:127.0.0.1:33017_solr] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2> 307987 INFO (jetty-launcher-1302-thread-4) [n:127.0.0.1:33017_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2> 308033 INFO (jetty-launcher-1302-thread-4) [n:127.0.0.1:33017_solr] o.a.s.c.ZkController Publish node=127.0.0.1:33017_solr as DOWN 2> 308039 INFO (jetty-launcher-1302-thread-4) [n:127.0.0.1:33017_solr] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33017_solr 2> 308050 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) 2> 308053 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) 2> 308060 WARN (jetty-launcher-1302-thread-4) [n:127.0.0.1:33017_solr] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2> 308574 INFO (jetty-launcher-1302-thread-2) [] o.a.s.c.CoreContainer Allowing use of paths: [_ALL_] 2> 308654 INFO (jetty-launcher-1302-thread-1) [n:127.0.0.1:33871_solr] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1 2> 308670 INFO (jetty-launcher-1302-thread-2) [] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33795/solr 2> 308673 INFO (jetty-launcher-1302-thread-2) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2> 308702 INFO (jetty-launcher-1302-thread-2) [n:127.0.0.1:40837_solr] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2> 308746 INFO (jetty-launcher-1302-thread-4) [n:127.0.0.1:33017_solr] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4 2> 308746 INFO (zkConnectionManagerCallback-1353-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected 2> 308749 INFO (jetty-launcher-1302-thread-2) [n:127.0.0.1:40837_solr] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2> 308768 WARN (jetty-launcher-1302-thread-2) [n:127.0.0.1:40837_solr] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2> 308772 INFO (jetty-launcher-1302-thread-2) [n:127.0.0.1:40837_solr] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2) 2> 308782 INFO (jetty-launcher-1302-thread-2) [n:127.0.0.1:40837_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2> 308796 INFO (jetty-launcher-1302-thread-2) [n:127.0.0.1:40837_solr] o.a.s.c.ZkController Publish node=127.0.0.1:40837_solr as DOWN 2> 308803 INFO (jetty-launcher-1302-thread-2) [n:127.0.0.1:40837_solr] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40837_solr 2> 308814 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2> 308814 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2> 308828 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2> 308833 WARN (jetty-launcher-1302-thread-2) [n:127.0.0.1:40837_solr] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2> 309024 INFO (jetty-launcher-1302-thread-4) [n:127.0.0.1:33017_solr] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2> 309025 INFO (jetty-launcher-1302-thread-1) [n:127.0.0.1:33871_solr] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2> 309145 INFO (jetty-launcher-1302-thread-1) [n:127.0.0.1:33871_solr] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33871, zkHost=127.0.0.1:33795/solr} 2> 309145 INFO (jetty-launcher-1302-thread-4) [n:127.0.0.1:33017_solr] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33017, zkHost=127.0.0.1:33795/solr} 2> 309149 INFO (jetty-launcher-1302-thread-2) [n:127.0.0.1:40837_solr] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node2 2> 309240 INFO (jetty-launcher-1302-thread-2) [n:127.0.0.1:40837_solr] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2> 309269 INFO (jetty-launcher-1302-thread-2) [n:127.0.0.1:40837_solr] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40837, zkHost=127.0.0.1:33795/solr} 2> 309838 INFO (jetty-launcher-1302-thread-3) [] o.a.s.c.CoreContainer Allowing use of paths: [_ALL_] 2> 309861 INFO (jetty-launcher-1302-thread-3) [] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33795/solr 2> 309862 INFO (jetty-launcher-1302-thread-3) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2> 309865 INFO (jetty-launcher-1302-thread-3) [n:127.0.0.1:37221_solr] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2> 309870 INFO (zkConnectionManagerCallback-1365-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected 2> 309871 INFO (jetty-launcher-1302-thread-3) [n:127.0.0.1:37221_solr] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2> 309881 WARN (jetty-launcher-1302-thread-3) [n:127.0.0.1:37221_solr] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2> 309884 INFO (jetty-launcher-1302-thread-3) [n:127.0.0.1:37221_solr] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3) 2> 309888 INFO (jetty-launcher-1302-thread-3) [n:127.0.0.1:37221_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2> 309893 INFO (jetty-launcher-1302-thread-3) [n:127.0.0.1:37221_solr] o.a.s.c.ZkController Publish node=127.0.0.1:37221_solr as DOWN 2> 309895 INFO (jetty-launcher-1302-thread-3) [n:127.0.0.1:37221_solr] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37221_solr 2> 309903 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4) 2> 309902 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4) 2> 309905 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4) 2> 309906 INFO (zkCallback-1364-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4) 2> 309914 WARN (jetty-launcher-1302-thread-3) [n:127.0.0.1:37221_solr] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2> 310181 INFO (jetty-launcher-1302-thread-3) [n:127.0.0.1:37221_solr] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node3 2> 310265 INFO (jetty-launcher-1302-thread-3) [n:127.0.0.1:37221_solr] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2> 310291 INFO (jetty-launcher-1302-thread-3) [n:127.0.0.1:37221_solr] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37221, zkHost=127.0.0.1:33795/solr} 2> 310303 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.MiniSolrCloudCluster waitForAllNodes: numServers=4 2> 310303 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.MiniSolrCloudCluster waitForNode: 127.0.0.1:33871_solr 2> 310304 WARN (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 310312 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2> 310320 INFO (zkConnectionManagerCallback-1374-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected 2> 310321 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2> 310321 WARN (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 310323 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 310325 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33795/solr ready 2> 310326 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.MiniSolrCloudCluster waitForNode: 127.0.0.1:33017_solr 2> 310326 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.MiniSolrCloudCluster waitForNode: 127.0.0.1:40837_solr 2> 310326 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.MiniSolrCloudCluster waitForNode: 127.0.0.1:37221_solr 2> 310326 WARN (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 310333 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.ConnectionManager Waiting up to 45000ms for client to connect to ZooKeeper 2> 310337 INFO (zkConnectionManagerCallback-1376-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected 2> 310337 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2> 310337 WARN (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 310457 WARN (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 310462 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.ConnectionManager Waiting up to 45000ms for client to connect to ZooKeeper 2> 310468 INFO (zkConnectionManagerCallback-1378-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected 2> 310469 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2> 310469 WARN (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2> 310729 INFO (TEST-CollectionsAPIDistributedZkTest.addReplicaTest-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting addReplicaTest 2> 310815 INFO (OverseerThreadFactory-1341-thread-1) [] o.a.s.c.a.c.CreateCollectionCmd Create collection addReplicaColl 2> 310948 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"addReplicaColl_shard2_replica_n1", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "collection":"addReplicaColl", 2> "shard":"shard2", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 310955 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"addReplicaColl_shard1_replica_n2", 2> "node_name":"127.0.0.1:37221_solr", 2> "base_url":"http://127.0.0.1:37221/solr", 2> "collection":"addReplicaColl", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 310960 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"addReplicaColl_shard2_replica_n4", 2> "node_name":"127.0.0.1:33871_solr", 2> "base_url":"http://127.0.0.1:33871/solr", 2> "collection":"addReplicaColl", 2> "shard":"shard2", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 310966 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"addReplicaColl_shard1_replica_n6", 2> "node_name":"127.0.0.1:40837_solr", 2> "base_url":"http://127.0.0.1:40837/solr", 2> "collection":"addReplicaColl", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 311076 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 311162 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=addReplicaColl_shard1_replica_n2&action=CREATE&numShards=2&collection=addReplicaColl&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 311162 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr x:addReplicaColl_shard1_replica_n6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=addReplicaColl_shard1_replica_n6&action=CREATE&numShards=2&collection=addReplicaColl&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 311162 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr x:addReplicaColl_shard2_replica_n4] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=addReplicaColl_shard2_replica_n4&action=CREATE&numShards=2&collection=addReplicaColl&shard=shard2&wt=javabin&version=2&replicaType=NRT 2> 311162 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:addReplicaColl_shard2_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=addReplicaColl_shard2_replica_n1&action=CREATE&numShards=2&collection=addReplicaColl&shard=shard2&wt=javabin&version=2&replicaType=NRT 2> 311277 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 311277 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 311277 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 311277 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 311295 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.s.IndexSchema Schema name=minimal 2> 311295 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.s.IndexSchema Schema name=minimal 2> 311295 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.s.IndexSchema Schema name=minimal 2> 311295 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.s.IndexSchema Schema name=minimal 2> 311302 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 311302 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 311302 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 311302 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 311303 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'addReplicaColl_shard2_replica_n1' using configuration from configset conf, trusted=true 2> 311303 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'addReplicaColl_shard1_replica_n2' using configuration from configset conf, trusted=true 2> 311303 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'addReplicaColl_shard2_replica_n4' using configuration from configset conf, trusted=true 2> 311303 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'addReplicaColl_shard1_replica_n6' using configuration from configset conf, trusted=true 2> 311311 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node2/addReplicaColl_shard1_replica_n6], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node2/addReplicaColl_shard1_replica_n6/data/] 2> 311311 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/addReplicaColl_shard2_replica_n4], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/addReplicaColl_shard2_replica_n4/data/] 2> 311311 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/addReplicaColl_shard2_replica_n1], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/addReplicaColl_shard2_replica_n1/data/] 2> 311311 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node3/addReplicaColl_shard1_replica_n2], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node3/addReplicaColl_shard1_replica_n2/data/] 2> 311748 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2> 311748 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2> 311748 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2> 311748 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2> 311759 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 311759 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 311759 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 311759 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 311760 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 311760 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 311760 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 311760 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 311772 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 311772 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 311772 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 311772 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 311772 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 311772 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 311772 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 311772 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 311806 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 311806 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 311806 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 311806 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 311808 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 311808 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 311808 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 311808 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 311812 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 311812 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 311812 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 311812 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 311813 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936524285640704 2> 311813 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936524285640704 2> 311813 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936524285640704 2> 311813 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936524285640704 2> 311838 INFO (searcherExecutor-1385-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard2_replica_n1 addReplicaColl shard2 core_node3) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 311838 INFO (searcherExecutor-1389-thread-1-processing-127.0.0.1:33871_solr addReplicaColl_shard2_replica_n4 addReplicaColl shard2 core_node7) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 311838 INFO (searcherExecutor-1384-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n2 addReplicaColl shard1 core_node5) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 311838 INFO (searcherExecutor-1387-thread-1-processing-127.0.0.1:40837_solr addReplicaColl_shard1_replica_n6 addReplicaColl shard1 core_node8) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 311852 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/addReplicaColl/terms/shard1 to Terms{values={core_node5=0}, version=0} for registerTerm 2> 311852 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.c.ZkShardTerms Successful update of terms at /collections/addReplicaColl/terms/shard2 to Terms{values={core_node7=0}, version=0} for registerTerm 2> 311853 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.c.ZkShardTerms Failed to save terms, version is not a match, retrying 2> 311853 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.c.ZkShardTerms Failed to save terms, version is not a match, retrying 2> 311854 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/addReplicaColl/leaders/shard1 2> 311854 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/addReplicaColl/leaders/shard2 2> 311857 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/addReplicaColl/terms/shard1 to Terms{values={core_node8=0, core_node5=0}, version=1} for registerTerm 2> 311858 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/addReplicaColl/terms/shard2 to Terms{values={core_node7=0, core_node3=0}, version=1} for registerTerm 2> 311858 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/addReplicaColl/leaders/shard1 2> 311858 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/addReplicaColl/leaders/shard2 2> 311876 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 311876 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 311877 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 311877 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 311877 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:37221/solr/addReplicaColl_shard1_replica_n2/ 2> 311877 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33871/solr/addReplicaColl_shard2_replica_n4/ 2> 311881 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=addReplicaColl_shard1_replica_n2 url=http://127.0.0.1:37221/solr START replicas=[http://127.0.0.1:40837/solr/addReplicaColl_shard1_replica_n6/] nUpdates=100 2> 311881 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.u.PeerSync PeerSync: core=addReplicaColl_shard2_replica_n4 url=http://127.0.0.1:33871/solr START replicas=[http://127.0.0.1:33017/solr/addReplicaColl_shard2_replica_n1/] nUpdates=100 2> 311886 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=addReplicaColl_shard1_replica_n2 url=http://127.0.0.1:37221/solr DONE. We have no versions. sync failed. 2> 311887 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.u.PeerSync PeerSync: core=addReplicaColl_shard2_replica_n4 url=http://127.0.0.1:33871/solr DONE. We have no versions. sync failed. 2> 311900 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.c.S.Request webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=4 2> 311900 INFO (qtp772667146-2532) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.c.S.Request webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=2 2> 311927 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate 2> 311927 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate 2> 311928 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway 2> 311928 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway 2> 311928 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/addReplicaColl/leaders/shard1/leader after winning as /collections/addReplicaColl/leader_elect/shard1/election/72085948270444554-core_node5-n_0000000000 2> 311928 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/addReplicaColl/leaders/shard2/leader after winning as /collections/addReplicaColl/leader_elect/shard2/election/72085948270444551-core_node7-n_0000000000 2> 311937 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33871/solr/addReplicaColl_shard2_replica_n4/ shard2 2> 311937 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:37221/solr/addReplicaColl_shard1_replica_n2/ shard1 2> 312045 INFO (zkCallback-1364-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312045 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312045 INFO (zkCallback-1328-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312045 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312050 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 312050 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 312069 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=addReplicaColl_shard1_replica_n2&action=CREATE&numShards=2&collection=addReplicaColl&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=919 2> 312069 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=addReplicaColl_shard2_replica_n4&action=CREATE&numShards=2&collection=addReplicaColl&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=919 2> 312161 INFO (zkCallback-1328-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312161 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312161 INFO (zkCallback-1364-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312161 INFO (zkCallback-1364-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312161 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312161 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312894 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=addReplicaColl_shard2_replica_n1&action=CREATE&numShards=2&collection=addReplicaColl&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1744 2> 312904 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=addReplicaColl_shard1_replica_n6&action=CREATE&numShards=2&collection=addReplicaColl&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1754 2> 312924 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2> 312999 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312999 INFO (zkCallback-1364-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312999 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312999 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312999 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312999 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312999 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312999 INFO (zkCallback-1364-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312999 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 312999 INFO (zkCallback-1328-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313002 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=addReplicaColl&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2205 2> 313017 INFO (TEST-CollectionsAPIDistributedZkTest.addReplicaTest-seed#[DF064544E215B325]) [] o.a.s.c.MiniSolrCloudCluster waitForActiveCollection: addReplicaColl 2> 313042 INFO (OverseerThreadFactory-1341-thread-2) [ c:addReplicaColl s:shard1] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:37221_solr for creating new replica of shard shard1 for collection addReplicaColl 2> 313042 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 313046 INFO (OverseerThreadFactory-1341-thread-2) [ c:addReplicaColl s:shard1] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command. 2> 313051 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"addReplicaColl_shard1_replica_n9", 2> "node_name":"127.0.0.1:37221_solr", 2> "base_url":"http://127.0.0.1:37221/solr", 2> "collection":"addReplicaColl", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"addreplica"} 2> 313158 INFO (zkCallback-1364-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313158 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313158 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313158 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313158 INFO (zkCallback-1364-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313158 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313158 INFO (zkCallback-1328-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313158 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313158 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313158 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313167 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n9] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node10&collection.configName=conf&name=addReplicaColl_shard1_replica_n9&action=CREATE&collection=addReplicaColl&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 313194 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 313198 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.s.IndexSchema Schema name=minimal 2> 313199 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 313201 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.CoreContainer Creating SolrCore 'addReplicaColl_shard1_replica_n9' using configuration from configset conf, trusted=true 2> 313203 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node3/addReplicaColl_shard1_replica_n9], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node3/addReplicaColl_shard1_replica_n9/data/] 2> 313280 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313280 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313280 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313280 INFO (zkCallback-1364-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313280 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313280 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313280 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313280 INFO (zkCallback-1328-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313280 INFO (zkCallback-1364-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313280 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313306 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 313306 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 313312 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 313312 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 313322 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 313323 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 313326 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 313327 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936525873184768 2> 313335 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.ZkShardTerms Successful update of terms at /collections/addReplicaColl/terms/shard1 to Terms{values={core_node10=0, core_node8=0, core_node5=0}, version=2} for registerTerm 2> 313335 INFO (searcherExecutor-1407-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 313340 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/addReplicaColl/leaders/shard1 2> 313345 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.ZkController Core needs to recover:addReplicaColl_shard1_replica_n9 2> 313346 INFO (updateExecutor-1361-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.u.DefaultSolrCoreState Running recovery 2> 313351 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true 2> 313351 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy startupVersions is empty 2> 313353 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node10&collection.configName=conf&name=addReplicaColl_shard1_replica_n9&action=CREATE&collection=addReplicaColl&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=187 2> 313362 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr c:addReplicaColl] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:37221_solr&action=ADDREPLICA&collection=addReplicaColl&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=329 2> 313376 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.S.Request webapp=/solr path=/admin/ping params={wt=javabin&version=2} hits=0 status=0 QTime=16 2> 313377 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.S.Request webapp=/solr path=/admin/ping params={wt=javabin&version=2} status=0 QTime=17 2> 313382 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[addReplicaColl_shard1_replica_n9] 2> 313384 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null} 2> 313385 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy Publishing state of core [addReplicaColl_shard1_replica_n9] as recovering, leader is [http://127.0.0.1:37221/solr/addReplicaColl_shard1_replica_n2/] and I am [http://127.0.0.1:37221/solr/addReplicaColl_shard1_replica_n9/] 2> 313390 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:37221/solr]; [WaitForState: action=PREPRECOVERY&core=addReplicaColl_shard1_replica_n2&nodeName=127.0.0.1:37221_solr&coreNodeName=core_node10&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true] 2> 313395 INFO (qtp1052031273-2522) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n2] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node10, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true 2> 313396 INFO (qtp1052031273-2522) [n:127.0.0.1:37221_solr] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=addReplicaColl, shard=shard1, thisCore=addReplicaColl_shard1_replica_n2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:37221_solr, coreNodeName=core_node10, onlyIfActiveCheckResult=false, nodeProps: core_node10:{ 2> "core":"addReplicaColl_shard1_replica_n9", 2> "node_name":"127.0.0.1:37221_solr", 2> "base_url":"http://127.0.0.1:37221/solr", 2> "state":"down", 2> "type":"NRT", 2> "force_set_state":"false"} 2> 313396 INFO (qtp1052031273-2522) [n:127.0.0.1:37221_solr] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=addReplicaColl, shard=shard1, thisCore=addReplicaColl_shard1_replica_n2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:37221_solr, coreNodeName=core_node10, onlyIfActiveCheckResult=false, nodeProps: core_node10:{ 2> "core":"addReplicaColl_shard1_replica_n9", 2> "node_name":"127.0.0.1:37221_solr", 2> "base_url":"http://127.0.0.1:37221/solr", 2> "state":"down", 2> "type":"NRT", 2> "force_set_state":"false"} 2> 313396 INFO (qtp1052031273-2522) [n:127.0.0.1:37221_solr] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=addReplicaColl, shard=shard1, thisCore=addReplicaColl_shard1_replica_n2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:37221_solr, coreNodeName=core_node10, onlyIfActiveCheckResult=false, nodeProps: core_node10:{ 2> "core":"addReplicaColl_shard1_replica_n9", 2> "node_name":"127.0.0.1:37221_solr", 2> "base_url":"http://127.0.0.1:37221/solr", 2> "state":"down", 2> "type":"NRT", 2> "force_set_state":"false"} 2> 313407 INFO (OverseerThreadFactory-1341-thread-3) [ c:addReplicaColl s:shard1] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:33017_solr for creating new replica of shard shard1 for collection addReplicaColl 2> 313407 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 313410 INFO (OverseerThreadFactory-1341-thread-3) [ c:addReplicaColl s:shard1] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command. 2> 313414 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"addReplicaColl_shard1_replica_n11", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "collection":"addReplicaColl", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"addreplica"} 2> 313521 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313521 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313521 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313521 INFO (zkCallback-1364-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313521 INFO (zkCallback-1328-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313521 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313521 INFO (zkCallback-1364-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313521 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313521 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313521 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313536 INFO (watches-1366-thread-2) [n:127.0.0.1:37221_solr] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=addReplicaColl, shard=shard1, thisCore=addReplicaColl_shard1_replica_n2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:37221_solr, coreNodeName=core_node10, onlyIfActiveCheckResult=false, nodeProps: core_node10:{ 2> "core":"addReplicaColl_shard1_replica_n9", 2> "node_name":"127.0.0.1:37221_solr", 2> "base_url":"http://127.0.0.1:37221/solr", 2> "state":"recovering", 2> "type":"NRT", 2> "force_set_state":"false"} 2> 313538 INFO (qtp1052031273-2522) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:37221_solr&onlyIfLeaderActive=true&core=addReplicaColl_shard1_replica_n2&coreNodeName=core_node10&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=143 2> 313554 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr x:addReplicaColl_shard1_replica_n11] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node12&collection.configName=conf&name=addReplicaColl_shard1_replica_n11&action=CREATE&collection=addReplicaColl&shard=shard1&wt=javabin&version=2&replicaType=NRT&property.instanceDir=/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-002 2> 313568 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 313572 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.s.IndexSchema Schema name=minimal 2> 313574 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 313574 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.CoreContainer Creating SolrCore 'addReplicaColl_shard1_replica_n11' using configuration from configset conf, trusted=true 2> 313577 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-002], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-002/data/] 2> 313667 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313668 INFO (zkCallback-1364-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313668 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313668 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313668 INFO (zkCallback-1328-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313667 INFO (zkCallback-1364-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313667 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313668 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313667 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313668 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313681 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 313681 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 313685 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 313685 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 313692 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 313693 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 313694 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 313695 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936526259060736 2> 313707 INFO (searcherExecutor-1414-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 313710 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.ZkShardTerms Successful update of terms at /collections/addReplicaColl/terms/shard1 to Terms{values={core_node10=0, core_node8=0, core_node12=0, core_node5=0}, version=3} for registerTerm 2> 313712 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/addReplicaColl/leaders/shard1 2> 313719 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.ZkController Core needs to recover:addReplicaColl_shard1_replica_n11 2> 313720 INFO (updateExecutor-1334-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.u.DefaultSolrCoreState Running recovery 2> 313722 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true 2> 313722 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.RecoveryStrategy startupVersions is empty 2> 313724 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node12&collection.configName=conf&name=addReplicaColl_shard1_replica_n11&action=CREATE&collection=addReplicaColl&shard=shard1&wt=javabin&version=2&replicaType=NRT&property.instanceDir=/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-002} status=0 QTime=172 2> 313733 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:addReplicaColl] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=ADDREPLICA&collection=addReplicaColl&shard=shard1&type=NRT&wt=javabin&version=2&property.instanceDir=/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-002} status=0 QTime=335 2> 313734 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.S.Request webapp=/solr path=/admin/ping params={wt=javabin&version=2} hits=0 status=0 QTime=3 2> 313735 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.S.Request webapp=/solr path=/admin/ping params={wt=javabin&version=2} status=0 QTime=3 2> 313736 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[addReplicaColl_shard1_replica_n11] 2> 313737 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null} 2> 313737 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.RecoveryStrategy Publishing state of core [addReplicaColl_shard1_replica_n11] as recovering, leader is [http://127.0.0.1:37221/solr/addReplicaColl_shard1_replica_n2/] and I am [http://127.0.0.1:33017/solr/addReplicaColl_shard1_replica_n11/] 2> 313741 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:37221/solr]; [WaitForState: action=PREPRECOVERY&core=addReplicaColl_shard1_replica_n2&nodeName=127.0.0.1:33017_solr&coreNodeName=core_node12&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true] 2> 313745 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n2] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node12, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true 2> 313746 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=addReplicaColl, shard=shard1, thisCore=addReplicaColl_shard1_replica_n2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:33017_solr, coreNodeName=core_node12, onlyIfActiveCheckResult=false, nodeProps: core_node12:{ 2> "core":"addReplicaColl_shard1_replica_n11", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "state":"down", 2> "type":"NRT", 2> "force_set_state":"false"} 2> 313746 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=addReplicaColl, shard=shard1, thisCore=addReplicaColl_shard1_replica_n2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:33017_solr, coreNodeName=core_node12, onlyIfActiveCheckResult=false, nodeProps: core_node12:{ 2> "core":"addReplicaColl_shard1_replica_n11", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "state":"down", 2> "type":"NRT", 2> "force_set_state":"false"} 2> 313747 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=addReplicaColl, shard=shard1, thisCore=addReplicaColl_shard1_replica_n2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:33017_solr, coreNodeName=core_node12, onlyIfActiveCheckResult=false, nodeProps: core_node12:{ 2> "core":"addReplicaColl_shard1_replica_n11", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "state":"down", 2> "type":"NRT", 2> "force_set_state":"false"} 2> 313765 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={core=addReplicaColl_shard1_replica_n11&action=STATUS&indexInfo=true&wt=javabin&version=2} status=0 QTime=20 2> 313787 INFO (OverseerThreadFactory-1341-thread-4) [ c:addReplicaColl s:shard1] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:33871_solr for creating new replica of shard shard1 for collection addReplicaColl 2> 313788 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 313789 ERROR (OverseerThreadFactory-1341-thread-4) [ c:addReplicaColl s:shard1] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: addReplicaColl operation: addreplica failed 2> => org.apache.solr.common.SolrException: Another replica with the same core name already exists for this collection 2> at org.apache.solr.cloud.api.collections.AddReplicaCmd.assignReplicaDetails(AddReplicaCmd.java:388) 2> org.apache.solr.common.SolrException: Another replica with the same core name already exists for this collection 2> at org.apache.solr.cloud.api.collections.AddReplicaCmd.assignReplicaDetails(AddReplicaCmd.java:388) ~[main/:?] 2> at org.apache.solr.cloud.api.collections.AddReplicaCmd.lambda$addReplica$0(AddReplicaCmd.java:181) ~[main/:?] 2> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?] 2> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?] 2> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?] 2> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?] 2> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?] 2> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?] 2> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?] 2> at org.apache.solr.cloud.api.collections.AddReplicaCmd.addReplica(AddReplicaCmd.java:183) ~[main/:?] 2> at org.apache.solr.cloud.api.collections.AddReplicaCmd.call(AddReplicaCmd.java:85) ~[main/:?] 2> at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:126) [main/:?] 2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:566) [main/:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289) [solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 313800 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr c:addReplicaColl] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=addReplicaColl_shard1_replica_n11&action=addreplica&collection=addReplicaColl&shard=shard1&wt=javabin&version=2} status=400 QTime=22 2> 313815 INFO (OverseerThreadFactory-1341-thread-5) [ c:addReplicaColl s:shard1] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:33871_solr for creating new replica of shard shard1 for collection addReplicaColl 2> 313816 INFO (OverseerThreadFactory-1341-thread-5) [ c:addReplicaColl s:shard1] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command. 2> 313816 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 313820 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"propertyDotName", 2> "node_name":"127.0.0.1:33871_solr", 2> "base_url":"http://127.0.0.1:33871/solr", 2> "collection":"addReplicaColl", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"addreplica"} 2> 313929 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313929 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313929 INFO (zkCallback-1364-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313929 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313929 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313929 INFO (zkCallback-1364-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313929 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313929 INFO (zkCallback-1328-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313929 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313929 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 313947 INFO (watches-1366-thread-2) [n:127.0.0.1:37221_solr] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=addReplicaColl, shard=shard1, thisCore=addReplicaColl_shard1_replica_n2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:33017_solr, coreNodeName=core_node12, onlyIfActiveCheckResult=false, nodeProps: core_node12:{ 2> "core":"addReplicaColl_shard1_replica_n11", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "state":"recovering", 2> "type":"NRT", 2> "force_set_state":"false"} 2> 313947 INFO (watches-1366-thread-1) [n:127.0.0.1:37221_solr] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=addReplicaColl, shard=shard1, thisCore=addReplicaColl_shard1_replica_n2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:33017_solr, coreNodeName=core_node12, onlyIfActiveCheckResult=false, nodeProps: core_node12:{ 2> "core":"addReplicaColl_shard1_replica_n11", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "state":"recovering", 2> "type":"NRT", 2> "force_set_state":"false"} 2> 313948 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:33017_solr&onlyIfLeaderActive=true&core=addReplicaColl_shard1_replica_n2&coreNodeName=core_node12&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=203 2> 313957 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr x:propertyDotName] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node13&collection.configName=conf&name=propertyDotName&action=CREATE&collection=addReplicaColl&shard=shard1&property.name=propertyDotName&wt=javabin&version=2&replicaType=NRT 2> 313976 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 313980 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.s.IndexSchema Schema name=minimal 2> 313982 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 313982 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.CoreContainer Creating SolrCore 'propertyDotName' using configuration from configset conf, trusted=true 2> 313985 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/propertyDotName], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/propertyDotName/data/] 2> 314051 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:37221/solr/addReplicaColl_shard1_replica_n2/] - recoveringAfterStartup=[true] 2> 314053 WARN (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.u.PeerSyncWithLeader no frame of reference to tell if we've missed updates 2> 314054 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy PeerSync Recovery was not successful - trying replication. 2> 314054 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy Starting Replication Recovery. 2> 314054 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy Attempting to replicate from [http://127.0.0.1:37221/solr/addReplicaColl_shard1_replica_n2/]. 2> 314070 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314070 INFO (zkCallback-1364-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314070 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314070 INFO (zkCallback-1364-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314071 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314070 INFO (zkCallback-1328-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314070 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314071 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314071 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314070 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314138 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 314138 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 314153 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard2 r:core_node3 x:addReplicaColl_shard2_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=false&commit=true&softCommit=false&distrib.from=http://127.0.0.1:33871/solr/addReplicaColl_shard2_replica_n4/&commit_end_point=replicas&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 16 2> 314156 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 314156 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 314157 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard2 r:core_node7 x:addReplicaColl_shard2_replica_n4] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={update.distrib=TOLEADER&waitSearcher=true&openSearcher=false&commit=true&softCommit=false&distrib.from=http://127.0.0.1:37221/solr/addReplicaColl_shard1_replica_n2/&commit_end_point=leaders&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 58 2> 314183 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 314185 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 314187 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 314187 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936526774960128 2> 314206 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.u.p.DistributedUpdateProcessor Ignoring commit while not ACTIVE - state: BUFFERING replay: false 2> 314206 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.u.p.DistributedUpdateProcessor Ignoring commit while not ACTIVE - state: BUFFERING replay: false 2> 314207 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=false&commit=true&softCommit=false&distrib.from=http://127.0.0.1:37221/solr/addReplicaColl_shard1_replica_n2/&commit_end_point=replicas&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 19 2> 314207 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=false&commit=true&softCommit=false&distrib.from=http://127.0.0.1:37221/solr/addReplicaColl_shard1_replica_n2/&commit_end_point=replicas&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 19 2> 314230 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:addReplicaColl s:shard1 r:core_node8 x:addReplicaColl_shard1_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=false&commit=true&softCommit=false&distrib.from=http://127.0.0.1:37221/solr/addReplicaColl_shard1_replica_n2/&commit_end_point=replicas&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 41 2> 314240 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={waitSearcher=true&openSearcher=false&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 172 2> 314240 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.ZkShardTerms Successful update of terms at /collections/addReplicaColl/terms/shard1 to Terms{values={core_node10=0, core_node8=0, core_node12=0, core_node13=0, core_node5=0}, version=4} for registerTerm 2> 314241 INFO (searcherExecutor-1421-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 314245 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/addReplicaColl/leaders/shard1 2> 314258 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.ZkController Core needs to recover:propertyDotName 2> 314258 INFO (updateExecutor-1325-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.u.DefaultSolrCoreState Running recovery 2> 314260 INFO (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true 2> 314261 INFO (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.RecoveryStrategy startupVersions is empty 2> 314262 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node13&collection.configName=conf&name=propertyDotName&action=CREATE&collection=addReplicaColl&shard=shard1&property.name=propertyDotName&wt=javabin&version=2&replicaType=NRT} status=0 QTime=306 2> 314263 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.S.Request webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=1 2> 314266 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.S.Request webapp=/solr path=/admin/ping params={wt=javabin&version=2} hits=0 status=0 QTime=1 2> 314266 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.h.IndexFetcher Leader's generation: 1 2> 314267 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node5 x:addReplicaColl_shard1_replica_n2] o.a.s.c.S.Request webapp=/solr path=/admin/ping params={wt=javabin&version=2} status=0 QTime=1 2> 314267 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.h.IndexFetcher Leader's version: 0 2> 314267 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.h.IndexFetcher Follower's generation: 1 2> 314267 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.h.IndexFetcher Follower's version: 0 2> 314267 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy No replay needed. 2> 314276 INFO (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[propertyDotName] 2> 314277 INFO (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null} 2> 314278 INFO (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.RecoveryStrategy Publishing state of core [propertyDotName] as recovering, leader is [http://127.0.0.1:37221/solr/addReplicaColl_shard1_replica_n2/] and I am [http://127.0.0.1:33871/solr/propertyDotName/] 2> 314285 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy Replication Recovery was successful. 2> 314286 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy Registering as Active after recovery. 2> 314294 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy Updating version bucket highest from index after successful recovery. 2> 314295 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936526888206336 2> 314295 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:addReplicaColl] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=ADDREPLICA&collection=addReplicaColl&shard=shard1&type=NRT&property.name=propertyDotName&wt=javabin&version=2} status=0 QTime=489 2> 314295 INFO (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:37221/solr]; [WaitForState: action=PREPRECOVERY&core=addReplicaColl_shard1_replica_n2&nodeName=127.0.0.1:33871_solr&coreNodeName=core_node13&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true] 2> 314297 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy Finished recovery process, successful=[true] msTimeTaken=945.0 2> 314300 INFO (recoveryExecutor-1363-thread-1-processing-127.0.0.1:37221_solr addReplicaColl_shard1_replica_n9 addReplicaColl shard1 core_node10) [n:127.0.0.1:37221_solr c:addReplicaColl s:shard1 r:core_node10 x:addReplicaColl_shard1_replica_n9] o.a.s.c.RecoveryStrategy Finished recovery process. recoveringAfterStartup=true msTimeTaken=949.0 2> 314306 INFO (qtp1052031273-2522) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n2] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node13, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true 2> 314307 INFO (qtp1052031273-2522) [n:127.0.0.1:37221_solr] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=addReplicaColl, shard=shard1, thisCore=addReplicaColl_shard1_replica_n2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:33871_solr, coreNodeName=core_node13, onlyIfActiveCheckResult=false, nodeProps: core_node13:{ 2> "core":"propertyDotName", 2> "node_name":"127.0.0.1:33871_solr", 2> "base_url":"http://127.0.0.1:33871/solr", 2> "state":"down", 2> "type":"NRT", 2> "force_set_state":"false"} 2> 314307 INFO (qtp1052031273-2522) [n:127.0.0.1:37221_solr] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=addReplicaColl, shard=shard1, thisCore=addReplicaColl_shard1_replica_n2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:33871_solr, coreNodeName=core_node13, onlyIfActiveCheckResult=false, nodeProps: core_node13:{ 2> "core":"propertyDotName", 2> "node_name":"127.0.0.1:33871_solr", 2> "base_url":"http://127.0.0.1:33871/solr", 2> "state":"down", 2> "type":"NRT", 2> "force_set_state":"false"} 2> 314308 INFO (qtp1052031273-2522) [n:127.0.0.1:37221_solr] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=addReplicaColl, shard=shard1, thisCore=addReplicaColl_shard1_replica_n2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:33871_solr, coreNodeName=core_node13, onlyIfActiveCheckResult=false, nodeProps: core_node13:{ 2> "core":"propertyDotName", 2> "node_name":"127.0.0.1:33871_solr", 2> "base_url":"http://127.0.0.1:33871/solr", 2> "state":"down", 2> "type":"NRT", 2> "force_set_state":"false"} 2> 314310 INFO (TEST-CollectionsAPIDistributedZkTest.addReplicaTest-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 314326 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000008 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 314328 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 314333 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr x:addReplicaColl_shard1_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.addReplicaColl.shard1.replica_n6 tag=null 2> 314334 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n9] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.addReplicaColl.shard1.replica_n9 tag=null 2> 314335 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr x:addReplicaColl_shard1_replica_n11] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.addReplicaColl.shard1.replica_n11 tag=null 2> 314335 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr x:propertyDotName] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.addReplicaColl.shard1.core_node13 tag=null 2> 314402 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:addReplicaColl_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.addReplicaColl.shard2.replica_n1 tag=null 2> 314402 WARN (qtp1260674627-2517) [n:127.0.0.1:33017_solr x:addReplicaColl_shard1_replica_n11] o.a.s.c.RecoveryStrategy Stopping recovery for core=[addReplicaColl_shard1_replica_n11] coreNodeName=[core_node12] 2> 314402 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314402 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314402 INFO (zkCallback-1364-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314402 INFO (zkCallback-1364-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314403 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.c.SolrCore Core addReplicaColl_shard1_replica_n11 is not yet closed, waiting 100 ms before checking again. 2> 314402 INFO (zkCallback-1328-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314402 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314402 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314402 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314402 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314402 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314432 WARN (watches-1366-thread-2) [] o.a.s.c.c.ZkStateReader Error on calling watcher 2> => org.apache.solr.common.SolrException: core not found:addReplicaColl_shard1_replica_n2 2> at org.apache.solr.handler.admin.PrepRecoveryOp.lambda$execute$0(PrepRecoveryOp.java:92) 2> org.apache.solr.common.SolrException: core not found:addReplicaColl_shard1_replica_n2 2> at org.apache.solr.handler.admin.PrepRecoveryOp.lambda$execute$0(PrepRecoveryOp.java:92) ~[main/:?] 2> at org.apache.solr.common.cloud.ZkStateReader.lambda$waitForState$14(ZkStateReader.java:1877) ~[solr-solrj-zookeeper-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.common.cloud.ZkStateReader$DocCollectionAndLiveNodesWatcherWrapper.onStateChanged(ZkStateReader.java:2439) ~[solr-solrj-zookeeper-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.common.cloud.ZkStateReader$Notification.run(ZkStateReader.java:2144) [solr-solrj-zookeeper-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289) [solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 314434 WARN (watches-1366-thread-3) [] o.a.s.c.c.ZkStateReader Error on calling watcher 2> => org.apache.solr.common.SolrException: core not found:addReplicaColl_shard1_replica_n2 2> at org.apache.solr.handler.admin.PrepRecoveryOp.lambda$execute$0(PrepRecoveryOp.java:92) 2> org.apache.solr.common.SolrException: core not found:addReplicaColl_shard1_replica_n2 2> at org.apache.solr.handler.admin.PrepRecoveryOp.lambda$execute$0(PrepRecoveryOp.java:92) ~[main/:?] 2> at org.apache.solr.common.cloud.ZkStateReader.lambda$waitForState$14(ZkStateReader.java:1877) ~[solr-solrj-zookeeper-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.common.cloud.ZkStateReader$DocCollectionAndLiveNodesWatcherWrapper.onStateChanged(ZkStateReader.java:2439) ~[solr-solrj-zookeeper-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.common.cloud.ZkStateReader$Notification.run(ZkStateReader.java:2144) [solr-solrj-zookeeper-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289) [solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 314434 WARN (watches-1366-thread-4) [] o.a.s.c.c.ZkStateReader Error on calling watcher 2> => org.apache.solr.common.SolrException: core not found:addReplicaColl_shard1_replica_n2 2> at org.apache.solr.handler.admin.PrepRecoveryOp.lambda$execute$0(PrepRecoveryOp.java:92) 2> org.apache.solr.common.SolrException: core not found:addReplicaColl_shard1_replica_n2 2> at org.apache.solr.handler.admin.PrepRecoveryOp.lambda$execute$0(PrepRecoveryOp.java:92) ~[main/:?] 2> at org.apache.solr.common.cloud.ZkStateReader.lambda$waitForState$14(ZkStateReader.java:1877) ~[solr-solrj-zookeeper-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.common.cloud.ZkStateReader$DocCollectionAndLiveNodesWatcherWrapper.onStateChanged(ZkStateReader.java:2439) ~[solr-solrj-zookeeper-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.common.cloud.ZkStateReader$Notification.run(ZkStateReader.java:2144) [solr-solrj-zookeeper-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289) [solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 314444 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr x:addReplicaColl_shard2_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.addReplicaColl.shard2.replica_n4 tag=null 2> 314445 WARN (qtp699896367-2525) [n:127.0.0.1:33871_solr x:propertyDotName] o.a.s.c.RecoveryStrategy Stopping recovery for core=[propertyDotName] coreNodeName=[core_node13] 2> 314451 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.c.SolrCore Core propertyDotName is not yet closed, waiting 100 ms before checking again. 2> 314455 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:37221/solr/addReplicaColl_shard1_replica_n2/] - recoveringAfterStartup=[true] 2> 314457 WARN (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.u.PeerSyncWithLeader no frame of reference to tell if we've missed updates 2> 314457 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.RecoveryStrategy PeerSync Recovery was not successful - trying replication. 2> 314457 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.RecoveryStrategy RecoveryStrategy has been closed 2> 314457 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.RecoveryStrategy Finished recovery process, successful=[false] msTimeTaken=735.0 2> 314457 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.RecoveryStrategy Finished recovery process. recoveringAfterStartup=true msTimeTaken=735.0 2> 314458 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@754ecc18 addReplicaColl_shard1_replica_n11 2> 314461 ERROR (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.RecoveryStrategy Error while trying to recover. core=propertyDotName 2> => java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://127.0.0.1:37221/solr 2> at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) 2> java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://127.0.0.1:37221/solr 2> at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?] 2> at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?] 2> at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:922) ~[main/:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:639) [main/:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:335) [main/:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:311) [main/:9.2.1 (not a git checkout) - builduser] 2> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:212) [metrics-core-4.2.17.jar:4.2.17] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289) [solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://127.0.0.1:37221/solr 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:757) ~[solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.lambda$httpUriRequest$0(HttpSolrClient.java:310) ~[solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> ... 5 more 2> Caused by: java.net.SocketException: Socket closed 2> at java.net.SocketInputStream.read(SocketInputStream.java:183) ~[?:?] 2> at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?] 2> at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.16.jar:4.4.16] 2> at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.16.jar:4.4.16] 2> at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[httpcore-4.4.16.jar:4.4.16] 2> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.14.jar:4.5.14] 2> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.14.jar:4.5.14] 2> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.16.jar:4.4.16] 2> at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.16.jar:4.4.16] 2> at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[httpclient-4.5.14.jar:4.5.14] 2> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.16.jar:4.4.16] 2> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.16.jar:4.4.16] 2> at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:142) ~[main/:?] 2> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.14.jar:4.5.14] 2> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.14.jar:4.5.14] 2> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.14.jar:4.5.14] 2> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.14.jar:4.5.14] 2> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.14.jar:4.5.14] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.14.jar:4.5.14] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.14.jar:4.5.14] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:602) ~[solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.lambda$httpUriRequest$0(HttpSolrClient.java:310) ~[solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> ... 5 more 2> 314476 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr x:addReplicaColl_shard1_replica_n6] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@63c91f27 addReplicaColl_shard1_replica_n6 2> 314476 INFO (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.RecoveryStrategy Recovery for core propertyDotName has been closed 2> 314477 INFO (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.RecoveryStrategy Finished recovery process, successful=[false] msTimeTaken=215.0 2> 314477 INFO (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.RecoveryStrategy Finished recovery process. recoveringAfterStartup=true msTimeTaken=216.0 2> 314477 INFO (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@62ae5c3 propertyDotName 2> 314477 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr x:addReplicaColl_shard1_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.addReplicaColl.shard1.replica_n6 tag=SolrCore@63c91f27 2> 314478 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr x:addReplicaColl_shard1_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.addReplicaColl.shard1.leader tag=SolrCore@63c91f27 2> 314480 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr x:addReplicaColl_shard1_replica_n6] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 314509 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/addReplicaColl/terms/shard1 to Terms{values={core_node10=0, core_node8=0, core_node13=0, core_node5=0}, version=5} for removeTerm 2> 314509 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr] o.a.s.c.ZkShardTerms Failed to save terms, version is not a match, retrying 2> 314512 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n9] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@4cc42296 addReplicaColl_shard1_replica_n9 2> 314512 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/addReplicaColl/terms/shard1 to Terms{values={core_node10=0, core_node13=0, core_node5=0}, version=6} for removeTerm 2> 314513 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.addReplicaColl.shard1.replica_n2 tag=null 2> 314514 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration. 2> 314528 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=addReplicaColl_shard1_replica_n11&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=194 2> 314528 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration. 2> 314533 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=addReplicaColl_shard1_replica_n6&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=201 2> 314555 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/addReplicaColl/terms/shard1 to Terms{values={core_node10=0, core_node5=0}, version=7} for removeTerm 2> 314558 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:addReplicaColl_shard2_replica_n1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@58a460c5 addReplicaColl_shard2_replica_n1 2> 314558 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.addReplicaColl.shard1.replica_n11 tag=SolrCore@754ecc18 2> 314558 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:addReplicaColl_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.addReplicaColl.shard2.replica_n1 tag=SolrCore@58a460c5 2> 314559 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.addReplicaColl.shard1.leader tag=SolrCore@754ecc18 2> 314559 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration. 2> 314559 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:addReplicaColl_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.addReplicaColl.shard2.leader tag=SolrCore@58a460c5 2> 314559 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:addReplicaColl_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 314568 WARN (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.c.RecoveryStrategy Stopping recovery for core=[addReplicaColl_shard1_replica_n11] coreNodeName=[core_node12] 2> 314570 INFO (recoveryExecutor-1336-thread-1-processing-127.0.0.1:33017_solr addReplicaColl_shard1_replica_n11 addReplicaColl shard1 core_node12) [n:127.0.0.1:33017_solr c:addReplicaColl s:shard1 r:core_node12 x:addReplicaColl_shard1_replica_n11] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 314599 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=propertyDotName&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=265 2> 314608 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/addReplicaColl/terms/shard2 to Terms{values={core_node7=0}, version=2} for removeTerm 2> 314624 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration. 2> 314627 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=addReplicaColl_shard2_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=292 2> 314642 INFO (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.addReplicaColl.shard1.core_node13 tag=SolrCore@62ae5c3 2> 314642 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr x:addReplicaColl_shard2_replica_n4] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@5d42a333 addReplicaColl_shard2_replica_n4 2> 314643 INFO (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.addReplicaColl.shard1.leader tag=SolrCore@62ae5c3 2> 314643 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr x:addReplicaColl_shard2_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.addReplicaColl.shard2.replica_n4 tag=SolrCore@5d42a333 2> 314643 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr x:addReplicaColl_shard2_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.addReplicaColl.shard2.leader tag=SolrCore@5d42a333 2> 314651 WARN (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.c.RecoveryStrategy Stopping recovery for core=[propertyDotName] coreNodeName=[core_node13] 2> 314652 INFO (recoveryExecutor-1327-thread-1-processing-127.0.0.1:33871_solr propertyDotName addReplicaColl shard1 core_node13) [n:127.0.0.1:33871_solr c:addReplicaColl s:shard1 r:core_node13 x:propertyDotName] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 314652 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr x:addReplicaColl_shard2_replica_n4] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 314679 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/addReplicaColl/terms/shard2 to Terms{values={}, version=3} for removeTerm 2> 314692 WARN (zkCallback-1337-thread-2) [] o.a.s.c.LeaderElector Our node is no longer in line to be leader 2> 314693 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=addReplicaColl_shard2_replica_n4&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=358 2> 314715 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n9] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.addReplicaColl.shard1.replica_n9 tag=SolrCore@4cc42296 2> 314715 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n2] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@5ec80153 addReplicaColl_shard1_replica_n2 2> 314715 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n9] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.addReplicaColl.shard1.leader tag=SolrCore@4cc42296 2> 314715 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.addReplicaColl.shard1.replica_n2 tag=SolrCore@5ec80153 2> 314716 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.addReplicaColl.shard1.leader tag=SolrCore@5ec80153 2> 314716 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n9] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 314716 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr x:addReplicaColl_shard1_replica_n2] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 314748 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/addReplicaColl/terms/shard1 to Terms{values={core_node10=0}, version=8} for removeTerm 2> 314753 WARN (zkCallback-1352-thread-2) [] o.a.s.c.LeaderElector Our node is no longer in line to be leader 2> 314755 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=addReplicaColl_shard1_replica_n2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=422 2> 314759 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/addReplicaColl/terms/shard1 to Terms{values={}, version=9} for removeTerm 2> 314760 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration. 2> 314761 WARN (zkCallback-1328-thread-4) [] o.a.s.c.LeaderElector Our node is no longer in line to be leader 2> 314761 WARN (zkCallback-1337-thread-2) [] o.a.s.c.LeaderElector Our node is no longer in line to be leader 2> 314762 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=addReplicaColl_shard1_replica_n9&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=429 2> 314874 INFO (zkCallback-1364-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314874 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314874 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314874 INFO (zkCallback-1364-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314875 INFO (zkCallback-1364-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314874 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314875 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314874 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314874 INFO (zkCallback-1328-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314876 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314876 INFO (zkCallback-1364-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314876 INFO (zkCallback-1328-thread-7) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/addReplicaColl/state.json] for collection [addReplicaColl] has occurred - updating... (live nodes size: [4]) 2> 314917 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=addReplicaColl&action=DELETE&wt=javabin&version=2} status=0 QTime=600 2> 314921 INFO (TEST-CollectionsAPIDistributedZkTest.addReplicaTest-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending addReplicaTest 2> 315241 INFO (TEST-CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting testDeleteNonExistentCollection 2> 315265 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000010 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 315267 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 315269 ERROR (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: unknown_collection operation: delete failed 2> => org.apache.solr.common.SolrException: Could not find collection : unknown_collection 2> at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:119) 2> org.apache.solr.common.SolrException: Could not find collection : unknown_collection 2> at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:119) ~[solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.cloud.api.collections.CollectionHandlingUtils.collectionCmd(CollectionHandlingUtils.java:410) ~[main/:?] 2> at org.apache.solr.cloud.api.collections.DeleteCollectionCmd.call(DeleteCollectionCmd.java:127) ~[main/:?] 2> at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:126) [main/:?] 2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:566) [main/:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289) [solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 315274 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=unknown_collection&action=DELETE&wt=javabin&version=2} status=400 QTime=19 2> 315288 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection acollectionafterbaddelete 2> 315288 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000012 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 315510 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"acollectionafterbaddelete_shard1_replica_n1", 2> "node_name":"127.0.0.1:33871_solr", 2> "base_url":"http://127.0.0.1:33871/solr", 2> "collection":"acollectionafterbaddelete", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 315516 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"acollectionafterbaddelete_shard1_replica_n2", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "collection":"acollectionafterbaddelete", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 315623 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 315632 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=acollectionafterbaddelete_shard1_replica_n1&action=CREATE&numShards=1&collection=acollectionafterbaddelete&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 315632 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=acollectionafterbaddelete_shard1_replica_n2&action=CREATE&numShards=1&collection=acollectionafterbaddelete&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 315662 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 315662 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 315666 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.s.IndexSchema Schema name=minimal 2> 315666 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.s.IndexSchema Schema name=minimal 2> 315668 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 315668 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'acollectionafterbaddelete_shard1_replica_n2' using configuration from configset conf, trusted=true 2> 315668 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 315668 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'acollectionafterbaddelete_shard1_replica_n1' using configuration from configset conf, trusted=true 2> 315670 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/acollectionafterbaddelete_shard1_replica_n2], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/acollectionafterbaddelete_shard1_replica_n2/data/] 2> 315670 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/acollectionafterbaddelete_shard1_replica_n1], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/acollectionafterbaddelete_shard1_replica_n1/data/] 2> 315797 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 315797 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 315804 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 315804 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 315808 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 315808 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 315815 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 315815 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 315817 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 315819 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 315823 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 315825 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936528492527616 2> 315829 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 315841 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 315843 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 315844 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936528512450560 2> 315845 INFO (searcherExecutor-1432-thread-1-processing-127.0.0.1:33871_solr acollectionafterbaddelete_shard1_replica_n1 acollectionafterbaddelete shard1 core_node3) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 315859 INFO (searcherExecutor-1430-thread-1-processing-127.0.0.1:33017_solr acollectionafterbaddelete_shard1_replica_n2 acollectionafterbaddelete shard1 core_node4) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 315865 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/acollectionafterbaddelete/terms/shard1 to Terms{values={core_node3=0}, version=0} for registerTerm 2> 315865 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/acollectionafterbaddelete/leaders/shard1 2> 315870 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/acollectionafterbaddelete/terms/shard1 to Terms{values={core_node3=0, core_node4=0}, version=1} for registerTerm 2> 315871 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/acollectionafterbaddelete/leaders/shard1 2> 315880 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 315880 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 315880 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33017/solr/acollectionafterbaddelete_shard1_replica_n2/ 2> 315883 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=acollectionafterbaddelete_shard1_replica_n2 url=http://127.0.0.1:33017/solr START replicas=[http://127.0.0.1:33871/solr/acollectionafterbaddelete_shard1_replica_n1/] nUpdates=100 2> 315886 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=acollectionafterbaddelete_shard1_replica_n2 url=http://127.0.0.1:33017/solr DONE. We have no versions. sync failed. 2> 315898 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:acollectionafterbaddelete s:shard1 r:core_node3 x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.c.S.Request webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=2 2> 315901 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate 2> 315901 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway 2> 315901 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/acollectionafterbaddelete/leaders/shard1/leader after winning as /collections/acollectionafterbaddelete/leader_elect/shard1/election/72085948270444552-core_node4-n_0000000000 2> 315910 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33017/solr/acollectionafterbaddelete_shard1_replica_n2/ shard1 2> 316016 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 316016 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 316021 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:acollectionafterbaddelete s:shard1 r:core_node4 x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 316037 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=acollectionafterbaddelete_shard1_replica_n2&action=CREATE&numShards=1&collection=acollectionafterbaddelete&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=407 2> 316134 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 316134 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 316134 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 316900 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=acollectionafterbaddelete_shard1_replica_n1&action=CREATE&numShards=1&collection=acollectionafterbaddelete&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1270 2> 316912 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2> 316996 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 316996 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 316996 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 316996 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 316996 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 317000 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=acollectionafterbaddelete&nrtReplicas=2&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=1720 2> 317005 INFO (TEST-CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection-seed#[DF064544E215B325]) [] o.a.s.c.SolrCloudTestCase waitForState (acollectionafterbaddelete): Collection creation after a bad delete failed 2> 317014 INFO (TEST-CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 317034 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000014 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 317035 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 317040 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.acollectionafterbaddelete.shard1.replica_n2 tag=null 2> 317040 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.acollectionafterbaddelete.shard1.replica_n1 tag=null 2> 317134 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@7ebf3d8 acollectionafterbaddelete_shard1_replica_n1 2> 317136 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.acollectionafterbaddelete.shard1.replica_n1 tag=SolrCore@7ebf3d8 2> 317136 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.acollectionafterbaddelete.shard1.leader tag=SolrCore@7ebf3d8 2> 317139 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr x:acollectionafterbaddelete_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 317159 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@2df3b8f4 acollectionafterbaddelete_shard1_replica_n2 2> 317160 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.acollectionafterbaddelete.shard1.replica_n2 tag=SolrCore@2df3b8f4 2> 317160 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.acollectionafterbaddelete.shard1.leader tag=SolrCore@2df3b8f4 2> 317164 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:acollectionafterbaddelete_shard1_replica_n2] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 317181 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/acollectionafterbaddelete/terms/shard1 to Terms{values={core_node4=0}, version=2} for removeTerm 2> 317184 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration. 2> 317186 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=acollectionafterbaddelete_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=146 2> 317186 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/acollectionafterbaddelete/terms/shard1 to Terms{values={}, version=3} for removeTerm 2> 317190 WARN (zkCallback-1328-thread-1) [] o.a.s.c.LeaderElector Our node is no longer in line to be leader 2> 317191 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=acollectionafterbaddelete_shard1_replica_n2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=152 2> 317304 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 317304 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 317304 INFO (zkCallback-1328-thread-7) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 317304 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 317304 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 317305 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 317304 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 317304 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 317304 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 317305 INFO (zkCallback-1337-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 317304 INFO (zkCallback-1328-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/acollectionafterbaddelete/state.json] for collection [acollectionafterbaddelete] has occurred - updating... (live nodes size: [4]) 2> 317351 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=acollectionafterbaddelete&action=DELETE&wt=javabin&version=2} status=0 QTime=325 2> 317355 INFO (TEST-CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending testDeleteNonExistentCollection 2> 317603 INFO (TEST-CollectionsAPIDistributedZkTest.testMissingRequiredParameters-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting testMissingRequiredParameters 2> 317615 ERROR (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Missing required parameter: name 2> => org.apache.solr.common.SolrException: Missing required parameter: name 2> at org.apache.solr.common.params.RequiredSolrParams.getParams(RequiredSolrParams.java:86) 2> org.apache.solr.common.SolrException: Missing required parameter: name 2> at org.apache.solr.common.params.RequiredSolrParams.getParams(RequiredSolrParams.java:86) ~[solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.handler.admin.CollectionsHandler.copy(CollectionsHandler.java:2112) ~[main/:?] 2> at org.apache.solr.handler.admin.CollectionsHandler.copy(CollectionsHandler.java:2127) ~[main/:?] 2> at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$0(CollectionsHandler.java:589) ~[main/:?] 2> at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1878) ~[main/:?] 2> at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:355) ~[main/:?] 2> at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:333) ~[main/:?] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:929) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:877) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:548) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:252) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:220) [main/:?] 2> at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257) [main/:?] 2> at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:215) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) [main/:?] 2> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:187) [solr-test-framework-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) [jetty-rewrite-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.Server.handle(Server.java:563) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) [jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) [jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) [jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) [jetty-util-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) [jetty-util-10.0.13.jar:10.0.13] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 317619 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CREATE&numShards=2&wt=javabin&version=2} status=400 QTime=6 2> 317625 INFO (TEST-CollectionsAPIDistributedZkTest.testMissingRequiredParameters-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 317629 INFO (TEST-CollectionsAPIDistributedZkTest.testMissingRequiredParameters-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending testMissingRequiredParameters 2> 317879 INFO (TEST-CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting deletePartiallyCreatedCollection 2> 317897 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection halfdeletedcollection 2> 317897 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000016 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 318108 WARN (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (halfdeletedcollection) without cores. 2> 318114 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2> 318119 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=halfdeletedcollection&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&wt=javabin&version=2} status=0 QTime=232 2> 318134 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000018 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 318136 INFO (OverseerThreadFactory-1341-thread-5) [ c:halfdeletedcollection s:shard1] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:33017_solr for creating new replica of shard shard1 for collection halfdeletedcollection 2> 318141 INFO (OverseerThreadFactory-1341-thread-5) [ c:halfdeletedcollection s:shard1] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command. 2> 318146 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"halfdeletedcollection_shard1_replica_n1", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "collection":"halfdeletedcollection", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"addreplica"} 2> 318253 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 318260 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr x:halfdeletedcollection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&dataDir=/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-002&collection.configName=conf&name=halfdeletedcollection_shard1_replica_n1&action=CREATE&collection=halfdeletedcollection&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 318290 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 318292 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.s.IndexSchema Schema name=minimal 2> 318293 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 318293 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'halfdeletedcollection_shard1_replica_n1' using configuration from configset conf, trusted=true 2> 318295 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/halfdeletedcollection_shard1_replica_n1], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-002/] 2> 318390 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 318390 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 318393 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 318393 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 318400 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 318401 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 318403 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 318403 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936531195756544 2> 318413 INFO (searcherExecutor-1446-thread-1-processing-127.0.0.1:33017_solr halfdeletedcollection_shard1_replica_n1 halfdeletedcollection shard1 core_node2) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 318421 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/halfdeletedcollection/terms/shard1 to Terms{values={core_node2=0}, version=0} for registerTerm 2> 318421 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/halfdeletedcollection/leaders/shard1 2> 318429 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 318429 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 318429 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33017/solr/halfdeletedcollection_shard1_replica_n1/ 2> 318430 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2> 318431 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:33017/solr/halfdeletedcollection_shard1_replica_n1/ has no replicas 2> 318431 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/halfdeletedcollection/leaders/shard1/leader after winning as /collections/halfdeletedcollection/leader_elect/shard1/election/72085948270444552-core_node2-n_0000000000 2> 318436 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33017/solr/halfdeletedcollection_shard1_replica_n1/ shard1 2> 318541 INFO (zkCallback-1337-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 318541 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 318543 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard1 r:core_node2 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 318554 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&dataDir=/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-002&collection.configName=conf&name=halfdeletedcollection_shard1_replica_n1&action=CREATE&collection=halfdeletedcollection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=295 2> 318567 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:halfdeletedcollection] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={dataDir=/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-002&action=ADDREPLICA&collection=halfdeletedcollection&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=441 2> 318583 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000020 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 318585 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 318591 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr x:halfdeletedcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.halfdeletedcollection.shard1.replica_n1 tag=null 2> 318651 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 318651 INFO (zkCallback-1337-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 318651 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 318685 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@672b30d5 halfdeletedcollection_shard1_replica_n1 2> 318686 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr x:halfdeletedcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.halfdeletedcollection.shard1.replica_n1 tag=SolrCore@672b30d5 2> 318686 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr x:halfdeletedcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.halfdeletedcollection.shard1.leader tag=SolrCore@672b30d5 2> 318688 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr x:halfdeletedcollection_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 318703 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/halfdeletedcollection/terms/shard1 to Terms{values={}, version=1} for removeTerm 2> 318721 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=halfdeletedcollection_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=131 2> 318832 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 318832 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 318832 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 318865 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=halfdeletedcollection&action=DELETE&wt=javabin&version=2} status=0 QTime=290 2> 318878 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=1 2> 318891 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection halfdeletedcollection 2> 318892 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000022 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 319281 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"halfdeletedcollection_shard1_replica_n1", 2> "node_name":"127.0.0.1:33871_solr", 2> "base_url":"http://127.0.0.1:33871/solr", 2> "collection":"halfdeletedcollection", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 319320 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"halfdeletedcollection_shard2_replica_n2", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "collection":"halfdeletedcollection", 2> "shard":"shard2", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 319489 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 319542 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr x:halfdeletedcollection_shard2_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=halfdeletedcollection_shard2_replica_n2&action=CREATE&numShards=2&collection=halfdeletedcollection&shard=shard2&wt=javabin&version=2&replicaType=NRT 2> 319554 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr x:halfdeletedcollection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=halfdeletedcollection_shard1_replica_n1&action=CREATE&numShards=2&collection=halfdeletedcollection&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 319703 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 319703 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 319706 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.s.IndexSchema Schema name=minimal 2> 319707 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.s.IndexSchema Schema name=minimal 2> 319712 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 319713 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 319714 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'halfdeletedcollection_shard2_replica_n2' using configuration from configset conf, trusted=true 2> 319715 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'halfdeletedcollection_shard1_replica_n1' using configuration from configset conf, trusted=true 2> 319725 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/halfdeletedcollection_shard1_replica_n1], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/halfdeletedcollection_shard1_replica_n1/data/] 2> 319725 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/halfdeletedcollection_shard2_replica_n2], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/halfdeletedcollection_shard2_replica_n2/data/] 2> 319992 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 319996 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 319997 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 319996 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 320010 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 320010 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 320012 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 320013 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 325114 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 325133 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 325172 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 325173 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 325195 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 325195 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 325200 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936538322927616 2> 325201 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936538323976192 2> 325258 INFO (searcherExecutor-1454-thread-1-processing-127.0.0.1:33871_solr halfdeletedcollection_shard1_replica_n1 halfdeletedcollection shard1 core_node3) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 325268 INFO (searcherExecutor-1452-thread-1-processing-127.0.0.1:33017_solr halfdeletedcollection_shard2_replica_n2 halfdeletedcollection shard2 core_node4) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 325326 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/halfdeletedcollection/terms/shard2 to Terms{values={core_node4=0}, version=0} for registerTerm 2> 325327 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/halfdeletedcollection/terms/shard1 to Terms{values={core_node3=0}, version=0} for registerTerm 2> 325353 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/halfdeletedcollection/leaders/shard1 2> 325354 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/halfdeletedcollection/leaders/shard2 2> 325404 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 325404 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 325405 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 325405 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 325406 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33871/solr/halfdeletedcollection_shard1_replica_n1/ 2> 325407 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33017/solr/halfdeletedcollection_shard2_replica_n2/ 2> 325412 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2> 325412 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2> 325413 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.SyncStrategy http://127.0.0.1:33017/solr/halfdeletedcollection_shard2_replica_n2/ has no replicas 2> 325413 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:33871/solr/halfdeletedcollection_shard1_replica_n1/ has no replicas 2> 325414 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/halfdeletedcollection/leaders/shard1/leader after winning as /collections/halfdeletedcollection/leader_elect/shard1/election/72085948270444551-core_node3-n_0000000000 2> 325414 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/halfdeletedcollection/leaders/shard2/leader after winning as /collections/halfdeletedcollection/leader_elect/shard2/election/72085948270444552-core_node4-n_0000000000 2> 325436 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33871/solr/halfdeletedcollection_shard1_replica_n1/ shard1 2> 325436 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33017/solr/halfdeletedcollection_shard2_replica_n2/ shard2 2> 325451 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 325451 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 325451 INFO (zkCallback-1337-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 325451 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 325460 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfdeletedcollection s:shard2 r:core_node4 x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 325576 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 325576 INFO (zkCallback-1337-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 325576 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 325576 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 325576 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 325584 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfdeletedcollection s:shard1 r:core_node3 x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 325695 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 325695 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 325695 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 325695 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 325695 INFO (zkCallback-1337-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 325695 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 334556 ERROR (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Error from shard: http://127.0.0.1:33871/solr 2> => org.apache.solr.client.solrj.SolrServerException: Total timeout 15000 ms elapsed 2> at org.apache.solr.client.solrj.impl.Http2SolrClient$1.onFailure(Http2SolrClient.java:486) 2> org.apache.solr.client.solrj.SolrServerException: Total timeout 15000 ms elapsed 2> at org.apache.solr.client.solrj.impl.Http2SolrClient$1.onFailure(Http2SolrClient.java:486) ~[solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.eclipse.jetty.client.ResponseNotifier.notifyFailure(ResponseNotifier.java:192) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.ResponseNotifier.notifyFailure(ResponseNotifier.java:184) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpReceiver.abort(HttpReceiver.java:544) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpChannel.abortResponse(HttpChannel.java:149) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpChannel.abort(HttpChannel.java:142) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpExchange.abort(HttpExchange.java:273) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpConversation.abort(HttpConversation.java:159) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpRequest.abort(HttpRequest.java:865) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpConnection$RequestTimeouts.onExpired(HttpConnection.java:331) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpConnection$RequestTimeouts.onExpired(HttpConnection.java:311) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.CyclicTimeouts.onTimeoutExpired(CyclicTimeouts.java:106) ~[jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.CyclicTimeouts$Timeouts.onTimeoutExpired(CyclicTimeouts.java:193) ~[jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.CyclicTimeout$Wakeup.run(CyclicTimeout.java:294) ~[jetty-io-10.0.13.jar:10.0.13] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> Caused by: java.util.concurrent.TimeoutException: Total timeout 15000 ms elapsed 2> ... 11 more 2> 334618 ERROR (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Error from shard: http://127.0.0.1:33017/solr 2> => org.apache.solr.client.solrj.SolrServerException: Total timeout 15000 ms elapsed 2> at org.apache.solr.client.solrj.impl.Http2SolrClient$1.onFailure(Http2SolrClient.java:486) 2> org.apache.solr.client.solrj.SolrServerException: Total timeout 15000 ms elapsed 2> at org.apache.solr.client.solrj.impl.Http2SolrClient$1.onFailure(Http2SolrClient.java:486) ~[solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.eclipse.jetty.client.ResponseNotifier.notifyFailure(ResponseNotifier.java:192) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.ResponseNotifier.notifyFailure(ResponseNotifier.java:184) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpReceiver.abort(HttpReceiver.java:544) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpChannel.abortResponse(HttpChannel.java:149) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpChannel.abort(HttpChannel.java:142) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpExchange.abort(HttpExchange.java:273) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpConversation.abort(HttpConversation.java:159) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpRequest.abort(HttpRequest.java:865) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpConnection$RequestTimeouts.onExpired(HttpConnection.java:331) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.client.HttpConnection$RequestTimeouts.onExpired(HttpConnection.java:311) ~[jetty-client-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.CyclicTimeouts.onTimeoutExpired(CyclicTimeouts.java:106) ~[jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.CyclicTimeouts$Timeouts.onTimeoutExpired(CyclicTimeouts.java:193) ~[jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.CyclicTimeout$Wakeup.run(CyclicTimeout.java:294) ~[jetty-io-10.0.13.jar:10.0.13] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> Caused by: java.util.concurrent.TimeoutException: Total timeout 15000 ms elapsed 2> ... 11 more 2> 334635 ERROR (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Cleaning up collection [halfdeletedcollection]. 2> 334652 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 341328 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr x:halfdeletedcollection_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.halfdeletedcollection.shard2.replica_n2 tag=null 2> 341328 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr x:halfdeletedcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.halfdeletedcollection.shard1.replica_n1 tag=null 2> 341342 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=halfdeletedcollection_shard2_replica_n2&action=CREATE&numShards=2&collection=halfdeletedcollection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=21805 2> 341343 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=halfdeletedcollection_shard1_replica_n1&action=CREATE&numShards=2&collection=halfdeletedcollection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=21805 2> 341419 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr x:halfdeletedcollection_shard2_replica_n2] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@d75f218 halfdeletedcollection_shard2_replica_n2 2> 341421 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr x:halfdeletedcollection_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.halfdeletedcollection.shard2.replica_n2 tag=SolrCore@d75f218 2> 341422 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr x:halfdeletedcollection_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.halfdeletedcollection.shard2.leader tag=SolrCore@d75f218 2> 341428 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr x:halfdeletedcollection_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 341451 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr x:halfdeletedcollection_shard1_replica_n1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@338a8691 halfdeletedcollection_shard1_replica_n1 2> 341452 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr x:halfdeletedcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.halfdeletedcollection.shard1.replica_n1 tag=SolrCore@338a8691 2> 341453 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr x:halfdeletedcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.halfdeletedcollection.shard1.leader tag=SolrCore@338a8691 2> 341456 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr x:halfdeletedcollection_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 341486 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/halfdeletedcollection/terms/shard1 to Terms{values={}, version=1} for removeTerm 2> 341487 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/halfdeletedcollection/terms/shard2 to Terms{values={}, version=1} for removeTerm 2> 341495 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=halfdeletedcollection_shard2_replica_n2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=6786 2> 341495 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=halfdeletedcollection_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=6786 2> 341499 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 341499 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 341499 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 341499 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 341499 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 341499 INFO (zkCallback-1337-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 341630 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 341630 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 341630 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 341630 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 341630 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 341630 INFO (zkCallback-1328-thread-7) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 341630 INFO (zkCallback-1328-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfdeletedcollection/state.json] for collection [halfdeletedcollection] has occurred - updating... (live nodes size: [4]) 2> 341663 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Cleaned up artifacts for failed create collection for [halfdeletedcollection] 2> 341665 ERROR (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: halfdeletedcollection operation: create failed 2> => org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: halfdeletedcollection 2> at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:449) 2> org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: halfdeletedcollection 2> at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:449) ~[main/:?] 2> at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:126) [main/:?] 2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:566) [main/:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289) [solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 341673 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=halfdeletedcollection&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=400 QTime=22791 2> 341702 INFO (TEST-CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 341709 INFO (TEST-CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending deletePartiallyCreatedCollection > org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33017/solr: Underlying core creation failed while creating collection: halfdeletedcollection > at __randomizedtesting.SeedInfo.seed([DF064544E215B325:F58629A155CF93ED]:0) > at app//org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:745) > at app//org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259) > at app//org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:240) > at app//org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:386) > at app//org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:352) > at app//org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1184) > at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:890) > at app//org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:818) > at app//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:234) > at app//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:249) > at app//org.apache.solr.cloud.api.collections.AbstractCollectionsAPIDistributedZkTestBase.deletePartiallyCreatedCollection(AbstractCollectionsAPIDistributedZkTestBase.java:166) > at java.base@11.0.20/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at java.base@11.0.20/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at java.base@11.0.20/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base@11.0.20/java.lang.reflect.Method.invoke(Method.java:566) > at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) > at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) > at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) > at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) > at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) > at app//org.junit.rules.RunRules.evaluate(RunRules.java:20) > at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44) > at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) > at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) > at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) > at app//org.junit.rules.RunRules.evaluate(RunRules.java:20) > at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) > at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) > at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) > at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) > at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) > at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) > at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) > at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) > at app//org.junit.rules.RunRules.evaluate(RunRules.java:20) > at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) > at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) > at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) > at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) > at app//org.junit.rules.RunRules.evaluate(RunRules.java:20) > at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) > at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) > at java.base@11.0.20/java.lang.Thread.run(Thread.java:829) 2> NOTE: reproduce with: gradlew test --tests CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection -Dtests.seed=DF064544E215B325 -Dtests.badapples=true -Dtests.locale=fo-FO -Dtests.timezone=America/Argentina/Buenos_Aires -Dtests.asserts=true -Dtests.file.encoding=UTF-8 2> 342105 INFO (TEST-CollectionsAPIDistributedZkTest.testZeroNumShards-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting testZeroNumShards 2> 342169 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection acollection 2> 342170 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000024 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 342170 ERROR (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: acollection operation: create failed 2> => org.apache.solr.common.SolrException: numShards must be > 0 2> at org.apache.solr.cloud.api.collections.CreateCollectionCmd.populateShardNames(CreateCollectionCmd.java:601) 2> org.apache.solr.common.SolrException: numShards must be > 0 2> at org.apache.solr.cloud.api.collections.CreateCollectionCmd.populateShardNames(CreateCollectionCmd.java:601) ~[main/:?] 2> at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:139) ~[main/:?] 2> at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:126) [main/:?] 2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:566) [main/:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289) [solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 342176 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=10&collection.configName=conf&name=acollection&action=CREATE&numShards=0&wt=javabin&version=2} status=400 QTime=21 2> 342183 INFO (TEST-CollectionsAPIDistributedZkTest.testZeroNumShards-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 342186 INFO (TEST-CollectionsAPIDistributedZkTest.testZeroNumShards-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending testZeroNumShards 2> 342398 INFO (TEST-CollectionsAPIDistributedZkTest.testCollectionReload-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting testCollectionReload 2> 342416 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection reloaded_collection 2> 342417 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000026 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 342638 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"reloaded_collection_shard1_replica_n1", 2> "node_name":"127.0.0.1:40837_solr", 2> "base_url":"http://127.0.0.1:40837/solr", 2> "collection":"reloaded_collection", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 342642 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"reloaded_collection_shard2_replica_n2", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "collection":"reloaded_collection", 2> "shard":"shard2", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 342647 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"reloaded_collection_shard1_replica_n4", 2> "node_name":"127.0.0.1:37221_solr", 2> "base_url":"http://127.0.0.1:37221/solr", 2> "collection":"reloaded_collection", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 342652 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"reloaded_collection_shard2_replica_n6", 2> "node_name":"127.0.0.1:33871_solr", 2> "base_url":"http://127.0.0.1:33871/solr", 2> "collection":"reloaded_collection", 2> "shard":"shard2", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 342759 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 342783 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr x:reloaded_collection_shard2_replica_n6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=reloaded_collection_shard2_replica_n6&action=CREATE&numShards=2&collection=reloaded_collection&shard=shard2&wt=javabin&version=2&replicaType=NRT 2> 342783 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr x:reloaded_collection_shard2_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=reloaded_collection_shard2_replica_n2&action=CREATE&numShards=2&collection=reloaded_collection&shard=shard2&wt=javabin&version=2&replicaType=NRT 2> 342812 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr x:reloaded_collection_shard1_replica_n4] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=reloaded_collection_shard1_replica_n4&action=CREATE&numShards=2&collection=reloaded_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 342812 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr x:reloaded_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=reloaded_collection_shard1_replica_n1&action=CREATE&numShards=2&collection=reloaded_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 342846 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 342846 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 342846 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 342846 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 342858 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.s.IndexSchema Schema name=minimal 2> 342858 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.s.IndexSchema Schema name=minimal 2> 342858 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.s.IndexSchema Schema name=minimal 2> 342858 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.s.IndexSchema Schema name=minimal 2> 342871 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 342872 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 342871 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 342873 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'reloaded_collection_shard2_replica_n6' using configuration from configset conf, trusted=true 2> 342873 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'reloaded_collection_shard2_replica_n2' using configuration from configset conf, trusted=true 2> 342873 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'reloaded_collection_shard1_replica_n1' using configuration from configset conf, trusted=true 2> 342871 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 342873 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'reloaded_collection_shard1_replica_n4' using configuration from configset conf, trusted=true 2> 342884 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/reloaded_collection_shard2_replica_n2], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/reloaded_collection_shard2_replica_n2/data/] 2> 342884 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node3/reloaded_collection_shard1_replica_n4], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node3/reloaded_collection_shard1_replica_n4/data/] 2> 342884 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/reloaded_collection_shard2_replica_n6], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/reloaded_collection_shard2_replica_n6/data/] 2> 342884 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node2/reloaded_collection_shard1_replica_n1], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node2/reloaded_collection_shard1_replica_n1/data/] 2> 343104 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 343104 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 343104 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 343104 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 343107 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 343107 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 343107 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 343108 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 343113 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 343113 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 343113 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 343114 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 343114 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 343113 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 343114 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 343114 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 343148 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 343148 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 343148 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 343149 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 343150 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 343150 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 343150 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 343150 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 343153 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 343153 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 343153 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 343153 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 343155 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936557150109696 2> 343155 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936557150109696 2> 343155 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936557150109696 2> 343155 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936557150109696 2> 343185 INFO (searcherExecutor-1473-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 reloaded_collection shard1 core_node3) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 343185 INFO (searcherExecutor-1474-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 reloaded_collection shard2 core_node5) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 343185 INFO (searcherExecutor-1470-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 reloaded_collection shard2 core_node8) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 343185 INFO (searcherExecutor-1468-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 reloaded_collection shard1 core_node7) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 343203 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/reloaded_collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm 2> 343203 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.c.ZkShardTerms Failed to save terms, version is not a match, retrying 2> 343203 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.c.ZkShardTerms Failed to save terms, version is not a match, retrying 2> 343203 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/reloaded_collection/terms/shard1 to Terms{values={core_node3=0}, version=0} for registerTerm 2> 343204 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/reloaded_collection/leaders/shard2 2> 343204 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/reloaded_collection/leaders/shard1 2> 343207 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.c.ZkShardTerms Successful update of terms at /collections/reloaded_collection/terms/shard1 to Terms{values={core_node3=0, core_node7=0}, version=1} for registerTerm 2> 343208 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/reloaded_collection/terms/shard2 to Terms{values={core_node8=0, core_node5=0}, version=1} for registerTerm 2> 343208 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/reloaded_collection/leaders/shard2 2> 343208 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/reloaded_collection/leaders/shard1 2> 343225 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 343225 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 343225 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 343225 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 343225 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40837/solr/reloaded_collection_shard1_replica_n1/ 2> 343225 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33017/solr/reloaded_collection_shard2_replica_n2/ 2> 343228 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.u.PeerSync PeerSync: core=reloaded_collection_shard2_replica_n2 url=http://127.0.0.1:33017/solr START replicas=[http://127.0.0.1:33871/solr/reloaded_collection_shard2_replica_n6/] nUpdates=100 2> 343228 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=reloaded_collection_shard1_replica_n1 url=http://127.0.0.1:40837/solr START replicas=[http://127.0.0.1:37221/solr/reloaded_collection_shard1_replica_n4/] nUpdates=100 2> 343232 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.u.PeerSync PeerSync: core=reloaded_collection_shard2_replica_n2 url=http://127.0.0.1:33017/solr DONE. We have no versions. sync failed. 2> 343235 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=reloaded_collection_shard1_replica_n1 url=http://127.0.0.1:40837/solr DONE. We have no versions. sync failed. 2> 343263 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:reloaded_collection s:shard2 r:core_node8 x:reloaded_collection_shard2_replica_n6] o.a.s.c.S.Request webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=10 2> 343263 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr c:reloaded_collection s:shard1 r:core_node7 x:reloaded_collection_shard1_replica_n4] o.a.s.c.S.Request webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=10 2> 343269 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate 2> 343269 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate 2> 343269 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway 2> 343269 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway 2> 343269 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/reloaded_collection/leaders/shard1/leader after winning as /collections/reloaded_collection/leader_elect/shard1/election/72085948270444553-core_node3-n_0000000000 2> 343270 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/reloaded_collection/leaders/shard2/leader after winning as /collections/reloaded_collection/leader_elect/shard2/election/72085948270444552-core_node5-n_0000000000 2> 343279 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33017/solr/reloaded_collection_shard2_replica_n2/ shard2 2> 343279 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40837/solr/reloaded_collection_shard1_replica_n1/ shard1 2> 343386 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 343386 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 343387 INFO (zkCallback-1352-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 343387 INFO (zkCallback-1352-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 343391 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:reloaded_collection s:shard2 r:core_node5 x:reloaded_collection_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 343391 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr c:reloaded_collection s:shard1 r:core_node3 x:reloaded_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 343417 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=reloaded_collection_shard1_replica_n1&action=CREATE&numShards=2&collection=reloaded_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=607 2> 343417 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=reloaded_collection_shard2_replica_n2&action=CREATE&numShards=2&collection=reloaded_collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=644 2> 343521 INFO (zkCallback-1352-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 343521 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 343520 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 343520 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 343520 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 343522 INFO (zkCallback-1352-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344265 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=reloaded_collection_shard2_replica_n6&action=CREATE&numShards=2&collection=reloaded_collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1493 2> 344283 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=reloaded_collection_shard1_replica_n4&action=CREATE&numShards=2&collection=reloaded_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1474 2> 344299 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344299 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344299 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344299 INFO (zkCallback-1352-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344299 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344299 INFO (zkCallback-1364-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344300 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344299 INFO (zkCallback-1352-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344299 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344300 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2> 344300 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344412 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344412 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344412 INFO (zkCallback-1352-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344412 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344412 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344412 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344412 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344413 INFO (zkCallback-1364-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344412 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344412 INFO (zkCallback-1352-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 344415 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=reloaded_collection&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2008 2> 344424 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000028 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 344437 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 344449 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344458 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 344468 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344485 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={async=61e066e3-13b2-47f6-a658-a4723c7e9677&name=reloaded_collection&action=RELOAD&wt=javabin&version=2} status=0 QTime=6 2> 344493 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=RELOAD, asyncId=61e066e3-13b2-47f6-a658-a4723c7e9677 2> 344496 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344501 INFO (qtp772667146-2532) [n:127.0.0.1:40837_solr x:reloaded_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={core=reloaded_collection_shard1_replica_n1&async=61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548&qt=/admin/cores&action=RELOAD&wt=javabin&version=2} status=0 QTime=1 2> 344501 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr x:reloaded_collection_shard1_replica_n4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={core=reloaded_collection_shard1_replica_n4&async=61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329&qt=/admin/cores&action=RELOAD&wt=javabin&version=2} status=0 QTime=1 2> 344503 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:reloaded_collection_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={core=reloaded_collection_shard2_replica_n2&async=61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603&qt=/admin/cores&action=RELOAD&wt=javabin&version=2} status=0 QTime=1 2> 344505 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr x:reloaded_collection_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={core=reloaded_collection_shard2_replica_n6&async=61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126&qt=/admin/cores&action=RELOAD&wt=javabin&version=2} status=0 QTime=1 2> 344507 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344514 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr] o.a.s.h.a.CoreAdminOperation Checking request status for : 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 2> 344526 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=12 2> 344529 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=13 2> 344537 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr x:reloaded_collection_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 344537 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr x:reloaded_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 344537 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr x:reloaded_collection_shard1_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 344538 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr x:reloaded_collection_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 344541 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr x:reloaded_collection_shard2_replica_n2] o.a.s.s.IndexSchema Schema name=minimal 2> 344541 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr x:reloaded_collection_shard2_replica_n6] o.a.s.s.IndexSchema Schema name=minimal 2> 344542 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr x:reloaded_collection_shard1_replica_n4] o.a.s.s.IndexSchema Schema name=minimal 2> 344542 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr x:reloaded_collection_shard1_replica_n1] o.a.s.s.IndexSchema Schema name=minimal 2> 344545 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr x:reloaded_collection_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 344545 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr x:reloaded_collection_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 344545 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr x:reloaded_collection_shard1_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 344545 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr x:reloaded_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 344546 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr x:reloaded_collection_shard2_replica_n2] o.a.s.c.CoreContainer Reloading SolrCore 'reloaded_collection_shard2_replica_n2' using configuration from configset conf 2> 344546 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr x:reloaded_collection_shard1_replica_n1] o.a.s.c.CoreContainer Reloading SolrCore 'reloaded_collection_shard1_replica_n1' using configuration from configset conf 2> 344546 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr x:reloaded_collection_shard1_replica_n4] o.a.s.c.CoreContainer Reloading SolrCore 'reloaded_collection_shard1_replica_n4' using configuration from configset conf 2> 344546 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr x:reloaded_collection_shard2_replica_n6] o.a.s.c.CoreContainer Reloading SolrCore 'reloaded_collection_shard2_replica_n6' using configuration from configset conf 2> 344579 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=10 2> 344638 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 344652 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 344670 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=8 2> 344716 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 344775 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 344784 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344794 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 344802 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344815 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 344827 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=2 2> 344842 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 344852 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344860 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344872 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344881 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344888 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344906 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344914 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344923 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 344931 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344941 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/reloaded_collection_shard2_replica_n2], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/reloaded_collection_shard2_replica_n2/data/] 2> 344943 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344955 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344964 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344967 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node2/reloaded_collection_shard1_replica_n1], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node2/reloaded_collection_shard1_replica_n1/data/] 2> 344971 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/reloaded_collection_shard2_replica_n6], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/reloaded_collection_shard2_replica_n6/data/] 2> 344972 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node3/reloaded_collection_shard1_replica_n4], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node3/reloaded_collection_shard1_replica_n4/data/] 2> 344974 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344985 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 344993 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 345001 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 345013 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 345023 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 345032 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 345038 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 345038 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 345045 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 345080 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 345082 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 345083 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 345084 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 345092 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936559181201408 2> 345141 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=14 2> 345142 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used. 2> 345154 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 345154 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 345154 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 345154 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 345164 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 345175 INFO (searcherExecutor-1496-thread-1-processing-127.0.0.1:33017_solr 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 345211 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@b529d62 reloaded_collection_shard2_replica_n2 2> 345216 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.reloaded_collection.shard2.replica_n2 tag=SolrCore@b529d62 2> 345221 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 345223 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 345225 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 345225 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 345238 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936559334293504 2> 345270 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 345270 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 345319 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used. 2> 345331 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 345366 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 345367 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 345368 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936559470608384 2> 345369 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=4 2> 345379 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 345401 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used. 2> 345411 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 345412 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 345413 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936559517794304 2> 345433 INFO (searcherExecutor-1496-thread-1-processing-127.0.0.1:33017_solr 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 345454 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used. 2> 345460 INFO (searcherExecutor-1492-thread-1-processing-127.0.0.1:37221_solr 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 345625 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@16af45f7 reloaded_collection_shard1_replica_n4 2> 345629 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.reloaded_collection.shard1.replica_n4 tag=SolrCore@16af45f7 2> 345642 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=4 2> 345642 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr] o.a.s.h.a.CoreAdminOperation Checking request status for : 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 2> 345642 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=3 2> 345660 INFO (searcherExecutor-1492-thread-1-processing-127.0.0.1:37221_solr 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 345662 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-1-processing-127.0.0.1:33017_solr reloaded_collection_shard2_replica_n2 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 reload) [n:127.0.0.1:33017_solr] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.reloaded_collection.shard2.leader tag=SolrCore@b529d62 2> 345662 INFO (searcherExecutor-1498-thread-1-processing-127.0.0.1:33871_solr 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 345663 INFO (searcherExecutor-1494-thread-1-processing-127.0.0.1:40837_solr 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 345664 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 345666 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@1e356215 reloaded_collection_shard1_replica_n1 2> 345666 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@6b5fbfd4 reloaded_collection_shard2_replica_n6 2> 345667 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.reloaded_collection.shard1.replica_n1 tag=SolrCore@1e356215 2> 345667 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.reloaded_collection.shard2.replica_n6 tag=SolrCore@6b5fbfd4 2> 345693 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 345699 INFO (searcherExecutor-1494-thread-1-processing-127.0.0.1:40837_solr 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 345701 INFO (searcherExecutor-1498-thread-1-processing-127.0.0.1:33871_solr 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 345707 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=0 2> 345721 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 345724 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-1-processing-127.0.0.1:37221_solr reloaded_collection_shard1_replica_n4 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 reload) [n:127.0.0.1:37221_solr] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.reloaded_collection.shard1.leader tag=SolrCore@16af45f7 2> 345743 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=11 2> 345752 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-1-processing-127.0.0.1:40837_solr reloaded_collection_shard1_replica_n1 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 reload) [n:127.0.0.1:40837_solr] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.reloaded_collection.shard1.leader tag=SolrCore@1e356215 2> 345774 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=3 2> 345778 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-1-processing-127.0.0.1:33871_solr reloaded_collection_shard2_replica_n6 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 reload) [n:127.0.0.1:33871_solr] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.reloaded_collection.shard2.leader tag=SolrCore@6b5fbfd4 2> 345811 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=javabin&version=2} status=0 QTime=1 2> 345820 INFO (TEST-CollectionsAPIDistributedZkTest.testCollectionReload-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 346654 INFO (qtp772667146-2532) [n:127.0.0.1:40837_solr] o.a.s.h.a.CoreAdminOperation Checking request status for : 61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548 2> 346655 INFO (qtp772667146-2532) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=61e066e3-13b2-47f6-a658-a4723c7e9677432689529927548&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=3 2> 346660 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr] o.a.s.h.a.CoreAdminOperation Checking request status for : 61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329 2> 346660 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=61e066e3-13b2-47f6-a658-a4723c7e9677432689533117329&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=1 2> 346665 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.h.a.CoreAdminOperation Checking request status for : 61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603 2> 346665 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=61e066e3-13b2-47f6-a658-a4723c7e9677432689534607603&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0 2> 346670 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr] o.a.s.h.a.CoreAdminOperation Checking request status for : 61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126 2> 346676 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=61e066e3-13b2-47f6-a658-a4723c7e9677432689536268126&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=6 2> 346688 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000030 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 347846 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000030 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 347847 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 347862 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr x:reloaded_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.reloaded_collection.shard1.replica_n1 tag=null 2> 347862 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr x:reloaded_collection_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.reloaded_collection.shard2.replica_n2 tag=null 2> 347862 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr x:reloaded_collection_shard1_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.reloaded_collection.shard1.replica_n4 tag=null 2> 347862 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr x:reloaded_collection_shard2_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.reloaded_collection.shard2.replica_n6 tag=null 2> 347928 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr x:reloaded_collection_shard2_replica_n6] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@59719c24 reloaded_collection_shard2_replica_n6 2> 347928 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr x:reloaded_collection_shard2_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.reloaded_collection.shard2.replica_n6 tag=SolrCore@59719c24 2> 347928 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr x:reloaded_collection_shard2_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.reloaded_collection.shard2.leader tag=SolrCore@59719c24 2> 347929 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr x:reloaded_collection_shard2_replica_n6] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 347948 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr x:reloaded_collection_shard1_replica_n1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@390fe75a reloaded_collection_shard1_replica_n1 2> 347948 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr x:reloaded_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.reloaded_collection.shard1.replica_n1 tag=SolrCore@390fe75a 2> 347948 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/reloaded_collection/terms/shard2 to Terms{values={core_node5=0}, version=2} for removeTerm 2> 347948 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr x:reloaded_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.reloaded_collection.shard1.leader tag=SolrCore@390fe75a 2> 347949 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr x:reloaded_collection_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 347949 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr x:reloaded_collection_shard2_replica_n2] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@3ba75b97 reloaded_collection_shard2_replica_n2 2> 347949 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr x:reloaded_collection_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.reloaded_collection.shard2.replica_n2 tag=SolrCore@3ba75b97 2> 347949 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration. 2> 347963 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr x:reloaded_collection_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.reloaded_collection.shard2.leader tag=SolrCore@3ba75b97 2> 347968 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr x:reloaded_collection_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 347992 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=reloaded_collection_shard2_replica_n6&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=132 2> 347996 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/reloaded_collection/terms/shard1 to Terms{values={core_node7=0}, version=2} for removeTerm 2> 347999 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/reloaded_collection/terms/shard2 to Terms{values={}, version=3} for removeTerm 2> 348001 INFO (zkCallback-1352-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348001 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348001 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348001 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348001 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348001 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348002 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348001 INFO (zkCallback-1352-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348001 INFO (zkCallback-1364-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348001 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348003 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=reloaded_collection_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=143 2> 348004 WARN (zkCallback-1328-thread-2) [] o.a.s.c.LeaderElector Our node is no longer in line to be leader 2> 348006 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=reloaded_collection_shard2_replica_n2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=146 2> 348032 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr x:reloaded_collection_shard1_replica_n4] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@31eb7b08 reloaded_collection_shard1_replica_n4 2> 348033 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr x:reloaded_collection_shard1_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.reloaded_collection.shard1.replica_n4 tag=SolrCore@31eb7b08 2> 348033 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr x:reloaded_collection_shard1_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.reloaded_collection.shard1.leader tag=SolrCore@31eb7b08 2> 348034 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr x:reloaded_collection_shard1_replica_n4] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 348048 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/reloaded_collection/terms/shard1 to Terms{values={}, version=3} for removeTerm 2> 348054 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration. 2> 348056 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=reloaded_collection_shard1_replica_n4&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=196 2> 348171 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348171 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348171 INFO (zkCallback-1328-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348171 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348171 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348171 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348171 INFO (zkCallback-1328-thread-7) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/reloaded_collection/state.json] for collection [reloaded_collection] has occurred - updating... (live nodes size: [4]) 2> 348218 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=reloaded_collection&action=DELETE&wt=javabin&version=2} status=0 QTime=2385 2> 348222 INFO (TEST-CollectionsAPIDistributedZkTest.testCollectionReload-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending testCollectionReload 2> 348482 INFO (TEST-CollectionsAPIDistributedZkTest.deleteCollectionRemovesStaleZkCollectionsNode-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting deleteCollectionRemovesStaleZkCollectionsNode 2> 348509 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000031 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 348517 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=out_of_sync_collection&action=DELETE&wt=javabin&version=2} status=0 QTime=17 2> 348522 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=0 2> 348527 INFO (TEST-CollectionsAPIDistributedZkTest.deleteCollectionRemovesStaleZkCollectionsNode-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 348531 INFO (TEST-CollectionsAPIDistributedZkTest.deleteCollectionRemovesStaleZkCollectionsNode-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending deleteCollectionRemovesStaleZkCollectionsNode 2> 348842 INFO (TEST-CollectionsAPIDistributedZkTest.testCreateNodeSet-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting testCreateNodeSet 2> 348862 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection nodeset_collection 2> 348863 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000033 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 349075 WARN (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (nodeset_collection) without cores. 2> 349079 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2> 349085 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=nodeset_collection&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=http://127.0.0.1:33871/solr,http://127.0.0.1:33017/solr&wt=javabin&version=2} status=0 QTime=233 2> 349090 INFO (TEST-CollectionsAPIDistributedZkTest.testCreateNodeSet-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 349103 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000035 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 349104 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 349220 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodeset_collection/state.json] for collection [nodeset_collection] has occurred - updating... (live nodes size: [4]) 2> 349241 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=nodeset_collection&action=DELETE&wt=javabin&version=2} status=0 QTime=145 2> 349245 INFO (TEST-CollectionsAPIDistributedZkTest.testCreateNodeSet-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending testCreateNodeSet 2> 349804 INFO (TEST-CollectionsAPIDistributedZkTest.testMissingNumShards-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting testMissingNumShards 2> 349900 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection acollection 2> 349901 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000037 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 349902 ERROR (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: acollection operation: create failed 2> => org.apache.solr.common.SolrException: numShards is a required param (when using CompositeId router). 2> at org.apache.solr.cloud.api.collections.CreateCollectionCmd.populateShardNames(CreateCollectionCmd.java:595) 2> org.apache.solr.common.SolrException: numShards is a required param (when using CompositeId router). 2> at org.apache.solr.cloud.api.collections.CreateCollectionCmd.populateShardNames(CreateCollectionCmd.java:595) ~[main/:?] 2> at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:139) ~[main/:?] 2> at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:126) [main/:?] 2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:566) [main/:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289) [solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 349956 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=10&collection.configName=conf&name=acollection&action=CREATE&wt=javabin&version=2} status=400 QTime=110 2> 349986 INFO (TEST-CollectionsAPIDistributedZkTest.testMissingNumShards-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 350022 INFO (TEST-CollectionsAPIDistributedZkTest.testMissingNumShards-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending testMissingNumShards 2> 350714 INFO (TEST-CollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting deleteCollectionOnlyInZk 2> 350738 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000039 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 350745 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=onlyinzk&action=DELETE&wt=javabin&version=2} status=0 QTime=16 2> 350750 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=0 2> 350763 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection onlyinzk 2> 350764 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000041 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 350986 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"onlyinzk_shard2_replica_n1", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "collection":"onlyinzk", 2> "shard":"shard2", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 350993 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"onlyinzk_shard1_replica_n2", 2> "node_name":"127.0.0.1:33871_solr", 2> "base_url":"http://127.0.0.1:33871/solr", 2> "collection":"onlyinzk", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 351100 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 351108 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr x:onlyinzk_shard2_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=onlyinzk_shard2_replica_n1&action=CREATE&numShards=2&collection=onlyinzk&shard=shard2&wt=javabin&version=2&replicaType=NRT 2> 351108 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr x:onlyinzk_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=onlyinzk_shard1_replica_n2&action=CREATE&numShards=2&collection=onlyinzk&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 351132 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 351132 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 351135 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.s.IndexSchema Schema name=minimal 2> 351135 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.s.IndexSchema Schema name=minimal 2> 351137 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 351137 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 351137 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'onlyinzk_shard2_replica_n1' using configuration from configset conf, trusted=true 2> 351137 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'onlyinzk_shard1_replica_n2' using configuration from configset conf, trusted=true 2> 351139 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/onlyinzk_shard1_replica_n2], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/onlyinzk_shard1_replica_n2/data/] 2> 351144 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/onlyinzk_shard2_replica_n1], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/onlyinzk_shard2_replica_n1/data/] 2> 351249 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 351250 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 351252 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 351253 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 351255 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 351255 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 351257 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 351257 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 351266 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 351266 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 351267 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 351267 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 351269 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 351269 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 351270 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936565659303936 2> 351270 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936565659303936 2> 351301 INFO (searcherExecutor-1522-thread-1-processing-127.0.0.1:33871_solr onlyinzk_shard1_replica_n2 onlyinzk shard1 core_node4) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 351303 INFO (searcherExecutor-1521-thread-1-processing-127.0.0.1:33017_solr onlyinzk_shard2_replica_n1 onlyinzk shard2 core_node3) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 351312 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/onlyinzk/terms/shard2 to Terms{values={core_node3=0}, version=0} for registerTerm 2> 351312 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/onlyinzk/terms/shard1 to Terms{values={core_node4=0}, version=0} for registerTerm 2> 351312 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/onlyinzk/leaders/shard2 2> 351312 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/onlyinzk/leaders/shard1 2> 351323 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 351323 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 351323 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 351323 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 351323 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33871/solr/onlyinzk_shard1_replica_n2/ 2> 351323 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33017/solr/onlyinzk_shard2_replica_n1/ 2> 351324 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2> 351324 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2> 351325 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.c.SyncStrategy http://127.0.0.1:33871/solr/onlyinzk_shard1_replica_n2/ has no replicas 2> 351325 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/onlyinzk/leaders/shard1/leader after winning as /collections/onlyinzk/leader_elect/shard1/election/72085948270444551-core_node4-n_0000000000 2> 351325 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:33017/solr/onlyinzk_shard2_replica_n1/ has no replicas 2> 351325 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/onlyinzk/leaders/shard2/leader after winning as /collections/onlyinzk/leader_elect/shard2/election/72085948270444552-core_node3-n_0000000000 2> 351333 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33017/solr/onlyinzk_shard2_replica_n1/ shard2 2> 351333 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33871/solr/onlyinzk_shard1_replica_n2/ shard1 2> 351446 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 351446 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 351446 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 351446 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 351452 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:onlyinzk s:shard2 r:core_node3 x:onlyinzk_shard2_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 351452 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:onlyinzk s:shard1 r:core_node4 x:onlyinzk_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 351496 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=onlyinzk_shard1_replica_n2&action=CREATE&numShards=2&collection=onlyinzk&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=388 2> 351505 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=onlyinzk_shard2_replica_n1&action=CREATE&numShards=2&collection=onlyinzk&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=398 2> 351526 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2> 351606 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 351606 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 351606 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 351606 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 351606 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 351606 INFO (zkCallback-1337-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 351613 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=onlyinzk&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=859 2> 351624 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=1 2> 351647 INFO (TEST-CollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 351703 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000043 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 351704 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 351710 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:onlyinzk_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.onlyinzk.shard2.replica_n1 tag=null 2> 351711 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr x:onlyinzk_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.onlyinzk.shard1.replica_n2 tag=null 2> 351828 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:onlyinzk_shard2_replica_n1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@36bac6c1 onlyinzk_shard2_replica_n1 2> 351831 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:onlyinzk_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.onlyinzk.shard2.replica_n1 tag=SolrCore@36bac6c1 2> 351833 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:onlyinzk_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.onlyinzk.shard2.leader tag=SolrCore@36bac6c1 2> 351846 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:onlyinzk_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 351931 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr x:onlyinzk_shard1_replica_n2] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@67e9df99 onlyinzk_shard1_replica_n2 2> 351934 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr x:onlyinzk_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.onlyinzk.shard1.replica_n2 tag=SolrCore@67e9df99 2> 351935 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr x:onlyinzk_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.onlyinzk.shard1.leader tag=SolrCore@67e9df99 2> 351941 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr x:onlyinzk_shard1_replica_n2] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 351960 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/onlyinzk/terms/shard2 to Terms{values={}, version=1} for removeTerm 2> 351960 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/onlyinzk/terms/shard1 to Terms{values={}, version=1} for removeTerm 2> 351967 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=onlyinzk_shard2_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=258 2> 351967 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=onlyinzk_shard1_replica_n2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=259 2> 352116 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 352117 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 352117 INFO (zkCallback-1328-thread-7) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 352116 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 352116 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 352116 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 352118 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 352118 INFO (zkCallback-1337-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 352118 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 352119 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 352118 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4]) 2> 352193 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=onlyinzk&action=DELETE&wt=javabin&version=2} status=0 QTime=510 2> 352197 INFO (TEST-CollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending deleteCollectionOnlyInZk 2> 352565 INFO (TEST-CollectionsAPIDistributedZkTest.testSpecificConfigsets-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting testSpecificConfigsets 2> 352597 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection withconfigset2 2> 352602 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000045 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 352839 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"withconfigset2_shard1_replica_n1", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "collection":"withconfigset2", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 352950 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/withconfigset2/state.json] for collection [withconfigset2] has occurred - updating... (live nodes size: [4]) 2> 352971 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr x:withconfigset2_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf2&newCollection=true&name=withconfigset2_shard1_replica_n1&action=CREATE&numShards=1&collection=withconfigset2&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 353203 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 353283 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.s.IndexSchema Schema name=minimal 2> 353314 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 353315 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'withconfigset2_shard1_replica_n1' using configuration from configset conf2, trusted=true 2> 353329 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/withconfigset2_shard1_replica_n1], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/withconfigset2_shard1_replica_n1/data/] 2> 353888 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2> 353920 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 353920 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 353944 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 353945 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 353983 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf2 2> 353987 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf2 2> 353990 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 353992 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936568513527808 2> 354022 INFO (searcherExecutor-1534-thread-1-processing-127.0.0.1:33017_solr withconfigset2_shard1_replica_n1 withconfigset2 shard1 core_node2) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 354048 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/withconfigset2/terms/shard1 to Terms{values={core_node2=0}, version=0} for registerTerm 2> 354050 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/withconfigset2/leaders/shard1 2> 354065 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 354065 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 354066 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33017/solr/withconfigset2_shard1_replica_n1/ 2> 354067 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2> 354068 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:33017/solr/withconfigset2_shard1_replica_n1/ has no replicas 2> 354068 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/withconfigset2/leaders/shard1/leader after winning as /collections/withconfigset2/leader_elect/shard1/election/72085948270444552-core_node2-n_0000000000 2> 354077 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33017/solr/withconfigset2_shard1_replica_n1/ shard1 2> 354186 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/withconfigset2/state.json] for collection [withconfigset2] has occurred - updating... (live nodes size: [4]) 2> 354186 INFO (zkCallback-1337-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/withconfigset2/state.json] for collection [withconfigset2] has occurred - updating... (live nodes size: [4]) 2> 354191 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:withconfigset2 s:shard1 r:core_node2 x:withconfigset2_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 354213 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf2&newCollection=true&name=withconfigset2_shard1_replica_n1&action=CREATE&numShards=1&collection=withconfigset2&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1254 2> 354261 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2> 354304 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/withconfigset2/state.json] for collection [withconfigset2] has occurred - updating... (live nodes size: [4]) 2> 354304 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/withconfigset2/state.json] for collection [withconfigset2] has occurred - updating... (live nodes size: [4]) 2> 354304 INFO (zkCallback-1337-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/withconfigset2/state.json] for collection [withconfigset2] has occurred - updating... (live nodes size: [4]) 2> 354307 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf2&name=withconfigset2&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=1729 2> 354333 INFO (TEST-CollectionsAPIDistributedZkTest.testSpecificConfigsets-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 354398 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000047 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 354401 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 354428 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:withconfigset2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.withconfigset2.shard1.replica_n1 tag=null 2> 354534 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:withconfigset2_shard1_replica_n1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@7833c451 withconfigset2_shard1_replica_n1 2> 354536 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:withconfigset2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.withconfigset2.shard1.replica_n1 tag=SolrCore@7833c451 2> 354536 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:withconfigset2_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.withconfigset2.shard1.leader tag=SolrCore@7833c451 2> 354544 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:withconfigset2_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 354567 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/withconfigset2/terms/shard1 to Terms{values={}, version=1} for removeTerm 2> 354572 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=withconfigset2_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=149 2> 354685 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/withconfigset2/state.json] for collection [withconfigset2] has occurred - updating... (live nodes size: [4]) 2> 354685 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/withconfigset2/state.json] for collection [withconfigset2] has occurred - updating... (live nodes size: [4]) 2> 354685 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/withconfigset2/state.json] for collection [withconfigset2] has occurred - updating... (live nodes size: [4]) 2> 354685 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/withconfigset2/state.json] for collection [withconfigset2] has occurred - updating... (live nodes size: [4]) 2> 354685 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/withconfigset2/state.json] for collection [withconfigset2] has occurred - updating... (live nodes size: [4]) 2> 354685 INFO (zkCallback-1337-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/withconfigset2/state.json] for collection [withconfigset2] has occurred - updating... (live nodes size: [4]) 2> 354685 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/withconfigset2/state.json] for collection [withconfigset2] has occurred - updating... (live nodes size: [4]) 2> 354735 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=withconfigset2&action=DELETE&wt=javabin&version=2} status=0 QTime=351 2> 354740 INFO (TEST-CollectionsAPIDistributedZkTest.testSpecificConfigsets-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending testSpecificConfigsets 2> 355140 INFO (TEST-CollectionsAPIDistributedZkTest.testNoConfigSetExist-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting testNoConfigSetExist 2> 355180 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection noconfig 2> 355181 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000049 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 355188 ERROR (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: noconfig operation: create failed 2> => org.apache.solr.common.SolrException: Can not find the specified config set: conf123 2> at org.apache.solr.cloud.api.collections.CollectionHandlingUtils.validateConfigOrThrowSolrException(CollectionHandlingUtils.java:387) 2> org.apache.solr.common.SolrException: Can not find the specified config set: conf123 2> at org.apache.solr.cloud.api.collections.CollectionHandlingUtils.validateConfigOrThrowSolrException(CollectionHandlingUtils.java:387) ~[main/:?] 2> at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:133) ~[main/:?] 2> at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:126) [main/:?] 2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:566) [main/:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289) [solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 355197 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf123&name=noconfig&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2} status=400 QTime=28 2> 356210 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=0 2> 356215 INFO (TEST-CollectionsAPIDistributedZkTest.testNoConfigSetExist-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 356221 INFO (TEST-CollectionsAPIDistributedZkTest.testNoConfigSetExist-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending testNoConfigSetExist 2> 356595 INFO (TEST-CollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting testCreateShouldFailOnExistingCore 2> 356621 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection halfcollectionblocker 2> 356623 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000051 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 356836 WARN (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (halfcollectionblocker) without cores. 2> 356842 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2> 356852 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=halfcollectionblocker&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&wt=javabin&version=2} status=0 QTime=246 2> 356870 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000053 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 356882 INFO (OverseerThreadFactory-1341-thread-5) [ c:halfcollectionblocker s:shard1] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:33871_solr for creating new replica of shard shard1 for collection halfcollectionblocker 2> 356885 INFO (OverseerThreadFactory-1341-thread-5) [ c:halfcollectionblocker s:shard1] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command. 2> 356892 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"halfcollection_shard1_replica_n1", 2> "node_name":"127.0.0.1:33871_solr", 2> "base_url":"http://127.0.0.1:33871/solr", 2> "collection":"halfcollectionblocker", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"addreplica"} 2> 357003 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker/state.json] for collection [halfcollectionblocker] has occurred - updating... (live nodes size: [4]) 2> 357011 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr x:halfcollection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node1&collection.configName=conf&name=halfcollection_shard1_replica_n1&action=CREATE&collection=halfcollectionblocker&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 357044 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 357048 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.s.IndexSchema Schema name=minimal 2> 357050 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 357051 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'halfcollection_shard1_replica_n1' using configuration from configset conf, trusted=true 2> 357054 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/halfcollection_shard1_replica_n1], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/halfcollection_shard1_replica_n1/data/] 2> 357185 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 357185 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 357201 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 357201 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 357213 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 357214 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 357216 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 357216 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936571894136832 2> 357233 INFO (searcherExecutor-1544-thread-1-processing-127.0.0.1:33871_solr halfcollection_shard1_replica_n1 halfcollectionblocker shard1 core_node1) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 357241 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/halfcollectionblocker/terms/shard1 to Terms{values={core_node1=0}, version=0} for registerTerm 2> 357241 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/halfcollectionblocker/leaders/shard1 2> 357250 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 357251 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 357251 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33871/solr/halfcollection_shard1_replica_n1/ 2> 357251 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2> 357252 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:33871/solr/halfcollection_shard1_replica_n1/ has no replicas 2> 357252 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/halfcollectionblocker/leaders/shard1/leader after winning as /collections/halfcollectionblocker/leader_elect/shard1/election/72085948270444551-core_node1-n_0000000000 2> 357258 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33871/solr/halfcollection_shard1_replica_n1/ shard1 2> 357363 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker/state.json] for collection [halfcollectionblocker] has occurred - updating... (live nodes size: [4]) 2> 357363 INFO (zkCallback-1328-thread-7) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker/state.json] for collection [halfcollectionblocker] has occurred - updating... (live nodes size: [4]) 2> 357366 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 357382 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node1&collection.configName=conf&name=halfcollection_shard1_replica_n1&action=CREATE&collection=halfcollectionblocker&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=372 2> 357392 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:halfcollectionblocker] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:33871_solr&name=halfcollection_shard1_replica_n1&action=ADDREPLICA&collection=halfcollectionblocker&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=533 2> 357408 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection halfcollectionblocker2 2> 357409 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000055 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 357525 INFO (zkCallback-1328-thread-7) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker/state.json] for collection [halfcollectionblocker] has occurred - updating... (live nodes size: [4]) 2> 357525 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker/state.json] for collection [halfcollectionblocker] has occurred - updating... (live nodes size: [4]) 2> 357525 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker/state.json] for collection [halfcollectionblocker] has occurred - updating... (live nodes size: [4]) 2> 357622 WARN (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (halfcollectionblocker2) without cores. 2> 357626 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2> 357633 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=halfcollectionblocker2&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&wt=javabin&version=2} status=0 QTime=233 2> 357651 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000057 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 357655 INFO (OverseerThreadFactory-1341-thread-5) [ c:halfcollectionblocker2 s:shard1] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:33017_solr for creating new replica of shard shard1 for collection halfcollectionblocker2 2> 357655 INFO (OverseerThreadFactory-1341-thread-5) [ c:halfcollectionblocker2 s:shard1] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command. 2> 357661 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"halfcollection_shard1_replica_n1", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "collection":"halfcollectionblocker2", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"addreplica"} 2> 357770 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker2/state.json] for collection [halfcollectionblocker2] has occurred - updating... (live nodes size: [4]) 2> 357779 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:halfcollection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node1&collection.configName=conf&name=halfcollection_shard1_replica_n1&action=CREATE&collection=halfcollectionblocker2&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 357822 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 357825 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.s.IndexSchema Schema name=minimal 2> 357827 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 357827 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'halfcollection_shard1_replica_n1' using configuration from configset conf, trusted=true 2> 357829 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/halfcollection_shard1_replica_n1], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/halfcollection_shard1_replica_n1/data/] 2> 357942 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 357943 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 357949 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 357949 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 357962 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 357963 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 357966 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 357967 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936572680568832 2> 357983 INFO (searcherExecutor-1550-thread-1-processing-127.0.0.1:33017_solr halfcollection_shard1_replica_n1 halfcollectionblocker2 shard1 core_node1) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 358000 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/halfcollectionblocker2/terms/shard1 to Terms{values={core_node1=0}, version=0} for registerTerm 2> 358000 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/halfcollectionblocker2/leaders/shard1 2> 358015 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 358015 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 358015 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33017/solr/halfcollection_shard1_replica_n1/ 2> 358017 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2> 358018 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:33017/solr/halfcollection_shard1_replica_n1/ has no replicas 2> 358019 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/halfcollectionblocker2/leaders/shard1/leader after winning as /collections/halfcollectionblocker2/leader_elect/shard1/election/72085948270444552-core_node1-n_0000000000 2> 358030 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33017/solr/halfcollection_shard1_replica_n1/ shard1 2> 358137 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker2/state.json] for collection [halfcollectionblocker2] has occurred - updating... (live nodes size: [4]) 2> 358137 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker2/state.json] for collection [halfcollectionblocker2] has occurred - updating... (live nodes size: [4]) 2> 358141 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr c:halfcollectionblocker2 s:shard1 r:core_node1 x:halfcollection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 358156 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node1&collection.configName=conf&name=halfcollection_shard1_replica_n1&action=CREATE&collection=halfcollectionblocker2&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=378 2> 358176 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr c:halfcollectionblocker2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:33017_solr&name=halfcollection_shard1_replica_n1&action=ADDREPLICA&collection=halfcollectionblocker2&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=533 2> 358206 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection halfcollection 2> 358207 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000059 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 358335 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker2/state.json] for collection [halfcollectionblocker2] has occurred - updating... (live nodes size: [4]) 2> 358334 INFO (zkCallback-1337-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker2/state.json] for collection [halfcollectionblocker2] has occurred - updating... (live nodes size: [4]) 2> 358335 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker2/state.json] for collection [halfcollectionblocker2] has occurred - updating... (live nodes size: [4]) 2> 358559 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"halfcollection_shard1_replica_n1", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "collection":"halfcollection", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 358667 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollection/state.json] for collection [halfcollection] has occurred - updating... (live nodes size: [4]) 2> 358679 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr x:halfcollection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=halfcollection_shard1_replica_n1&action=CREATE&numShards=1&collection=halfcollection&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 358681 WARN (qtp1260674627-2523) [n:127.0.0.1:33017_solr x:halfcollection_shard1_replica_n1] o.a.s.c.CoreContainer Creating a core with existing name is not allowed: 'halfcollection_shard1_replica_n1' 2> 358686 ERROR (qtp1260674627-2523) [n:127.0.0.1:33017_solr x:halfcollection_shard1_replica_n1] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Core with name 'halfcollection_shard1_replica_n1' already exists. 2> => org.apache.solr.common.SolrException: Core with name 'halfcollection_shard1_replica_n1' already exists. 2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1507) 2> org.apache.solr.common.SolrException: Core with name 'halfcollection_shard1_replica_n1' already exists. 2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1507) ~[main/:?] 2> at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:111) ~[main/:?] 2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:398) ~[main/:?] 2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:354) ~[main/:?] 2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:219) ~[main/:?] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:929) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:877) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:548) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:252) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:220) [main/:?] 2> at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257) [main/:?] 2> at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:215) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) [main/:?] 2> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:187) [solr-test-framework-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) [jetty-rewrite-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.Server.handle(Server.java:563) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:457) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) [jetty-util-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) [jetty-util-10.0.13.jar:10.0.13] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 358696 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr x:halfcollection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=halfcollection_shard1_replica_n1&action=CREATE&numShards=1&collection=halfcollection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=500 QTime=19 2> 358699 ERROR (qtp1260674627-2523) [n:127.0.0.1:33017_solr x:halfcollection_shard1_replica_n1] o.a.s.s.HttpSolrCall org.apache.solr.common.SolrException: Core with name 'halfcollection_shard1_replica_n1' already exists. 2> => org.apache.solr.common.SolrException: Core with name 'halfcollection_shard1_replica_n1' already exists. 2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1507) 2> org.apache.solr.common.SolrException: Core with name 'halfcollection_shard1_replica_n1' already exists. 2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1507) ~[main/:?] 2> at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:111) ~[main/:?] 2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:398) ~[main/:?] 2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:354) ~[main/:?] 2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:219) ~[main/:?] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224) ~[main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:929) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:877) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:548) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:252) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:220) [main/:?] 2> at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257) [main/:?] 2> at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:215) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) [main/:?] 2> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:187) [solr-test-framework-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) [jetty-rewrite-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:790) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.Server.handle(Server.java:563) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:457) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) [jetty-util-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) [jetty-util-10.0.13.jar:10.0.13] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 358715 ERROR (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Error from shard: http://127.0.0.1:33017/solr 2> => org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at null: Core with name 'halfcollection_shard1_replica_n1' already exists. 2> at org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:895) 2> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at null: Core with name 'halfcollection_shard1_replica_n1' already exists. 2> at org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:895) ~[solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:548) ~[solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.client.solrj.impl.Http2SolrClient$1.lambda$onHeaders$0(Http2SolrClient.java:461) ~[solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289) [solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 358724 ERROR (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Cleaning up collection [halfcollection]. 2> 358735 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 358745 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr x:halfcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.halfcollectionblocker2.shard1.core_node1 tag=null 2> 358843 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr x:halfcollection_shard1_replica_n1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@2a003f39 halfcollection_shard1_replica_n1 2> 358843 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr x:halfcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.halfcollectionblocker2.shard1.core_node1 tag=SolrCore@2a003f39 2> 358844 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr x:halfcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.halfcollectionblocker2.shard1.leader tag=SolrCore@2a003f39 2> 358846 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr x:halfcollection_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 358865 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/halfcollectionblocker2/terms/shard1 to Terms{values={}, version=1} for removeTerm 2> 358877 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=halfcollection_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=134 2> 359003 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker2/state.json] for collection [halfcollectionblocker2] has occurred - updating... (live nodes size: [4]) 2> 359003 INFO (zkCallback-1337-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker2/state.json] for collection [halfcollectionblocker2] has occurred - updating... (live nodes size: [4]) 2> 359003 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/halfcollectionblocker2/state.json] for collection [halfcollectionblocker2] has occurred - updating... (live nodes size: [4]) 2> 359005 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfcollection/state.json] for collection [halfcollection] has occurred - updating... (live nodes size: [4]) 2> 359005 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfcollection/state.json] for collection [halfcollection] has occurred - updating... (live nodes size: [4]) 2> 359044 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Cleaned up artifacts for failed create collection for [halfcollection] 2> 359045 ERROR (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: halfcollection operation: create failed 2> => org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: halfcollection 2> at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:449) 2> org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: halfcollection 2> at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:449) ~[main/:?] 2> at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:126) [main/:?] 2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:566) [main/:9.2.1 (not a git checkout) - builduser] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289) [solr-solrj-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 359059 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=halfcollection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:33871_solr,127.0.0.1:33017_solr&wt=javabin&version=2} status=400 QTime=865 2> 359083 INFO (TEST-CollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 359122 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000061 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 359128 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 359244 INFO (zkCallback-1328-thread-7) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfcollectionblocker2/state.json] for collection [halfcollectionblocker2] has occurred - updating... (live nodes size: [4]) 2> 359246 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfcollectionblocker2/state.json] for collection [halfcollectionblocker2] has occurred - updating... (live nodes size: [4]) 2> 359252 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfcollectionblocker2/state.json] for collection [halfcollectionblocker2] has occurred - updating... (live nodes size: [4]) 2> 359303 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=halfcollectionblocker2&action=DELETE&wt=javabin&version=2} status=0 QTime=193 2> 359315 ERROR (qtp1052031273-2522) [n:127.0.0.1:37221_solr] o.a.s.h.RequestHandlerBase org.apache.solr.cloud.ZkController$NotInClusterStateException: Timeout waiting for collection state. 2> => org.apache.solr.cloud.ZkController$NotInClusterStateException: Timeout waiting for collection state. 2> at org.apache.solr.handler.admin.PrepRecoveryOp.execute(PrepRecoveryOp.java:206) 2> org.apache.solr.cloud.ZkController$NotInClusterStateException: Timeout waiting for collection state. 2> at org.apache.solr.handler.admin.PrepRecoveryOp.execute(PrepRecoveryOp.java:206) ~[main/:?] 2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:398) ~[main/:?] 2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:354) ~[main/:?] 2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:219) ~[main/:?] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:929) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:877) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:548) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:252) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:220) [main/:?] 2> at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257) [main/:?] 2> at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:215) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) [main/:?] 2> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:187) [solr-test-framework-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) [jetty-rewrite-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.Server.handle(Server.java:563) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) [jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) [jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) [jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) [jetty-util-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) [jetty-util-10.0.13.jar:10.0.13] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 359324 INFO (qtp1052031273-2522) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:33871_solr&onlyIfLeaderActive=true&core=addReplicaColl_shard1_replica_n2&coreNodeName=core_node13&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=500 QTime=45019 2> 359327 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000063 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 359328 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 359325 ERROR (qtp1052031273-2522) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall org.apache.solr.cloud.ZkController$NotInClusterStateException: Timeout waiting for collection state. 2> => org.apache.solr.cloud.ZkController$NotInClusterStateException: Timeout waiting for collection state. 2> at org.apache.solr.handler.admin.PrepRecoveryOp.execute(PrepRecoveryOp.java:206) 2> org.apache.solr.cloud.ZkController$NotInClusterStateException: Timeout waiting for collection state. 2> at org.apache.solr.handler.admin.PrepRecoveryOp.execute(PrepRecoveryOp.java:206) ~[main/:?] 2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:398) ~[main/:?] 2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:354) ~[main/:?] 2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:219) ~[main/:?] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224) ~[main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:929) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:877) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:548) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:252) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:220) [main/:?] 2> at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257) [main/:?] 2> at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:215) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) [main/:?] 2> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:187) [solr-test-framework-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) [jetty-rewrite-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.Server.handle(Server.java:563) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) [jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) [jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) [jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) [jetty-util-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) [jetty-util-10.0.13.jar:10.0.13] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 359335 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr x:halfcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.halfcollectionblocker.shard1.core_node1 tag=null 2> 359393 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr x:halfcollection_shard1_replica_n1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@29f31827 halfcollection_shard1_replica_n1 2> 359393 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr x:halfcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.halfcollectionblocker.shard1.core_node1 tag=SolrCore@29f31827 2> 359394 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr x:halfcollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.halfcollectionblocker.shard1.leader tag=SolrCore@29f31827 2> 359396 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr x:halfcollection_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 359416 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/halfcollectionblocker/terms/shard1 to Terms{values={}, version=1} for removeTerm 2> 359421 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=halfcollection_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=87 2> 359549 INFO (zkCallback-1328-thread-7) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfcollectionblocker/state.json] for collection [halfcollectionblocker] has occurred - updating... (live nodes size: [4]) 2> 359550 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfcollectionblocker/state.json] for collection [halfcollectionblocker] has occurred - updating... (live nodes size: [4]) 2> 359549 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfcollectionblocker/state.json] for collection [halfcollectionblocker] has occurred - updating... (live nodes size: [4]) 2> 359549 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfcollectionblocker/state.json] for collection [halfcollectionblocker] has occurred - updating... (live nodes size: [4]) 2> 359549 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfcollectionblocker/state.json] for collection [halfcollectionblocker] has occurred - updating... (live nodes size: [4]) 2> 359550 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/halfcollectionblocker/state.json] for collection [halfcollectionblocker] has occurred - updating... (live nodes size: [4]) 2> 359589 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=halfcollectionblocker&action=DELETE&wt=javabin&version=2} status=0 QTime=275 2> 359593 INFO (TEST-CollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending testCreateShouldFailOnExistingCore 2> 359888 INFO (TEST-CollectionsAPIDistributedZkTest.testBadActionNames-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting testBadActionNames 2> 359902 ERROR (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Unknown action: BADACTION 2> => org.apache.solr.common.SolrException: Unknown action: BADACTION 2> at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:320) 2> org.apache.solr.common.SolrException: Unknown action: BADACTION 2> at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:320) ~[main/:?] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:929) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:877) [main/:?] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:548) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:252) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:220) [main/:?] 2> at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257) [main/:?] 2> at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:215) [main/:?] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) [main/:?] 2> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.apache.solr.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:187) [solr-test-framework-9.2.1.jar:9.2.1 (not a git checkout) - builduser] 2> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) [jetty-servlet-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) [jetty-rewrite-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.Server.handle(Server.java:563) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) [jetty-server-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) [jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) [jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) [jetty-io-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) [jetty-util-10.0.13.jar:10.0.13] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) [jetty-util-10.0.13.jar:10.0.13] 2> at java.lang.Thread.run(Thread.java:829) [?:?] 2> 359908 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=badactioncollection&action=BADACTION&numShards=2&wt=javabin&version=2} status=400 QTime=8 2> 359914 INFO (TEST-CollectionsAPIDistributedZkTest.testBadActionNames-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 359919 INFO (TEST-CollectionsAPIDistributedZkTest.testBadActionNames-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending testBadActionNames 2> 360296 INFO (TEST-CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting testCoresAreDistributedAcrossNodes 2> 360312 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection nodes_used_collection 2> 360313 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000065 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 360537 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"nodes_used_collection_shard2_replica_n1", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "collection":"nodes_used_collection", 2> "shard":"shard2", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 360543 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"nodes_used_collection_shard2_replica_n2", 2> "node_name":"127.0.0.1:33871_solr", 2> "base_url":"http://127.0.0.1:33871/solr", 2> "collection":"nodes_used_collection", 2> "shard":"shard2", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 360549 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"nodes_used_collection_shard1_replica_n4", 2> "node_name":"127.0.0.1:40837_solr", 2> "base_url":"http://127.0.0.1:40837/solr", 2> "collection":"nodes_used_collection", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 360556 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"nodes_used_collection_shard1_replica_n6", 2> "node_name":"127.0.0.1:37221_solr", 2> "base_url":"http://127.0.0.1:37221/solr", 2> "collection":"nodes_used_collection", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 360663 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 360672 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr x:nodes_used_collection_shard2_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard2_replica_n2&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2&replicaType=NRT 2> 360672 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr x:nodes_used_collection_shard2_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard2_replica_n1&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2&replicaType=NRT 2> 360673 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr x:nodes_used_collection_shard1_replica_n6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard1_replica_n6&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 360674 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr x:nodes_used_collection_shard1_replica_n4] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard1_replica_n4&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 360696 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 360696 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 360698 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 360698 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 360699 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.s.IndexSchema Schema name=minimal 2> 360699 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.s.IndexSchema Schema name=minimal 2> 360700 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.s.IndexSchema Schema name=minimal 2> 360701 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.s.IndexSchema Schema name=minimal 2> 360701 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 360701 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 360701 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard2_replica_n1' using configuration from configset conf, trusted=true 2> 360701 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 360701 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard1_replica_n4' using configuration from configset conf, trusted=true 2> 360702 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard2_replica_n2' using configuration from configset conf, trusted=true 2> 360703 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 360703 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard1_replica_n6' using configuration from configset conf, trusted=true 2> 360704 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node2/nodes_used_collection_shard1_replica_n4], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node2/nodes_used_collection_shard1_replica_n4/data/] 2> 360704 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/nodes_used_collection_shard2_replica_n1], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/nodes_used_collection_shard2_replica_n1/data/] 2> 360704 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/nodes_used_collection_shard2_replica_n2], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/nodes_used_collection_shard2_replica_n2/data/] 2> 360705 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node3/nodes_used_collection_shard1_replica_n6], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node3/nodes_used_collection_shard1_replica_n6/data/] 2> 360845 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 360845 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 360850 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 360850 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 360854 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 360855 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 360859 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 360859 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 360860 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 360861 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 360861 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 360861 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 360861 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 360862 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936575717244928 2> 360865 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 360884 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 360887 INFO (searcherExecutor-1562-thread-1-processing-127.0.0.1:40837_solr nodes_used_collection_shard1_replica_n4 nodes_used_collection shard1 core_node7) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 360897 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 360897 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 360899 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 360899 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 360900 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 360900 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 360902 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 360902 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936575759187968 2> 360903 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 360903 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936575760236544 2> 360903 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 360904 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 360931 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.ZkShardTerms Successful update of terms at /collections/nodes_used_collection/terms/shard1 to Terms{values={core_node7=0}, version=0} for registerTerm 2> 360932 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/nodes_used_collection/leaders/shard1 2> 360934 INFO (searcherExecutor-1561-thread-1-processing-127.0.0.1:33017_solr nodes_used_collection_shard2_replica_n1 nodes_used_collection shard2 core_node3) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 360934 INFO (searcherExecutor-1564-thread-1-processing-127.0.0.1:33871_solr nodes_used_collection_shard2_replica_n2 nodes_used_collection shard2 core_node5) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 360937 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 360939 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 360942 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 360943 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936575802179584 2> 360945 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/nodes_used_collection/terms/shard2 to Terms{values={core_node3=0}, version=0} for registerTerm 2> 360945 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.c.ZkShardTerms Failed to save terms, version is not a match, retrying 2> 360946 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/nodes_used_collection/leaders/shard2 2> 360946 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoute in=14998ms 2> 360956 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/nodes_used_collection/terms/shard2 to Terms{values={core_node3=0, core_node5=0}, version=1} for registerTerm 2> 360958 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/nodes_used_collection/leaders/shard2 2> 360959 INFO (searcherExecutor-1566-thread-1-processing-127.0.0.1:37221_solr nodes_used_collection_shard1_replica_n6 nodes_used_collection shard1 core_node8) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 360968 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 360969 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 360969 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33017/solr/nodes_used_collection_shard2_replica_n1/ 2> 360970 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/nodes_used_collection/terms/shard1 to Terms{values={core_node7=0, core_node8=0}, version=1} for registerTerm 2> 360971 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/nodes_used_collection/leaders/shard1 2> 360972 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.u.PeerSync PeerSync: core=nodes_used_collection_shard2_replica_n1 url=http://127.0.0.1:33017/solr START replicas=[http://127.0.0.1:33871/solr/nodes_used_collection_shard2_replica_n2/] nUpdates=100 2> 360974 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.u.PeerSync PeerSync: core=nodes_used_collection_shard2_replica_n1 url=http://127.0.0.1:33017/solr DONE. We have no versions. sync failed. 2> 361005 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:nodes_used_collection s:shard2 r:core_node5 x:nodes_used_collection_shard2_replica_n2] o.a.s.c.S.Request webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=9 2> 361010 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate 2> 361011 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway 2> 361011 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/nodes_used_collection/leaders/shard2/leader after winning as /collections/nodes_used_collection/leader_elect/shard2/election/72085948270444552-core_node3-n_0000000000 2> 361018 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33017/solr/nodes_used_collection_shard2_replica_n1/ shard2 2> 361124 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 361124 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 361137 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:nodes_used_collection s:shard2 r:core_node3 x:nodes_used_collection_shard2_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 361150 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard2_replica_n1&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=479 2> 361248 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 361248 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 361248 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 361449 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 361450 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 361450 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40837/solr/nodes_used_collection_shard1_replica_n4/ 2> 361452 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.u.PeerSync PeerSync: core=nodes_used_collection_shard1_replica_n4 url=http://127.0.0.1:40837/solr START replicas=[http://127.0.0.1:37221/solr/nodes_used_collection_shard1_replica_n6/] nUpdates=100 2> 361453 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.u.PeerSync PeerSync: core=nodes_used_collection_shard1_replica_n4 url=http://127.0.0.1:40837/solr DONE. We have no versions. sync failed. 2> 361465 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr c:nodes_used_collection s:shard1 r:core_node8 x:nodes_used_collection_shard1_replica_n6] o.a.s.c.S.Request webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=2 2> 361467 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate 2> 361468 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway 2> 361468 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/nodes_used_collection/leaders/shard1/leader after winning as /collections/nodes_used_collection/leader_elect/shard1/election/72085948270444553-core_node7-n_0000000000 2> 361474 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40837/solr/nodes_used_collection_shard1_replica_n4/ shard1 2> 361580 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 361580 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 361580 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 361580 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 361583 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr c:nodes_used_collection s:shard1 r:core_node7 x:nodes_used_collection_shard1_replica_n4] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 361597 INFO (qtp772667146-2520) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard1_replica_n4&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=925 2> 361693 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 361693 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 361693 INFO (zkCallback-1352-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 361693 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 361693 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 361993 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard2_replica_n2&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1323 2> 362007 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard1_replica_n6&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1335 2> 362019 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2> 362104 INFO (zkCallback-1352-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362104 INFO (zkCallback-1364-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362104 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362104 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362104 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362104 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362104 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362104 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362104 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362107 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=nodes_used_collection&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=1804 2> 362113 INFO (TEST-CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 362131 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000067 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 362132 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 362137 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr x:nodes_used_collection_shard1_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.nodes_used_collection.shard1.replica_n6 tag=null 2> 362137 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr x:nodes_used_collection_shard1_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.nodes_used_collection.shard1.replica_n4 tag=null 2> 362137 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:nodes_used_collection_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.nodes_used_collection.shard2.replica_n1 tag=null 2> 362139 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr x:nodes_used_collection_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.nodes_used_collection.shard2.replica_n2 tag=null 2> 362203 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:nodes_used_collection_shard2_replica_n1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@574a5edd nodes_used_collection_shard2_replica_n1 2> 362204 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:nodes_used_collection_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.nodes_used_collection.shard2.replica_n1 tag=SolrCore@574a5edd 2> 362205 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:nodes_used_collection_shard2_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.nodes_used_collection.shard2.leader tag=SolrCore@574a5edd 2> 362209 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr x:nodes_used_collection_shard1_replica_n4] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@a58f0d0 nodes_used_collection_shard1_replica_n4 2> 362213 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr x:nodes_used_collection_shard2_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 362215 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr x:nodes_used_collection_shard1_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.nodes_used_collection.shard1.replica_n4 tag=SolrCore@a58f0d0 2> 362221 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr x:nodes_used_collection_shard1_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.nodes_used_collection.shard1.leader tag=SolrCore@a58f0d0 2> 362232 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr x:nodes_used_collection_shard1_replica_n4] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 362261 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr x:nodes_used_collection_shard2_replica_n2] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@6c13c1cd nodes_used_collection_shard2_replica_n2 2> 362263 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr x:nodes_used_collection_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.nodes_used_collection.shard2.replica_n2 tag=SolrCore@6c13c1cd 2> 362264 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr x:nodes_used_collection_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.nodes_used_collection.shard2.leader tag=SolrCore@6c13c1cd 2> 362268 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr x:nodes_used_collection_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 362276 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/nodes_used_collection/terms/shard2 to Terms{values={core_node5=0}, version=2} for removeTerm 2> 362293 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr x:nodes_used_collection_shard1_replica_n6] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@288ddc40 nodes_used_collection_shard1_replica_n6 2> 362295 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr x:nodes_used_collection_shard1_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.nodes_used_collection.shard1.replica_n6 tag=SolrCore@288ddc40 2> 362295 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr x:nodes_used_collection_shard1_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.nodes_used_collection.shard1.leader tag=SolrCore@288ddc40 2> 362298 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr x:nodes_used_collection_shard1_replica_n6] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 362311 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/nodes_used_collection/terms/shard1 to Terms{values={core_node8=0}, version=2} for removeTerm 2> 362334 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr] o.a.s.c.ZkShardTerms Failed to save terms, version is not a match, retrying 2> 362335 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=nodes_used_collection_shard2_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=199 2> 362336 INFO (qtp772667146-2530) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=nodes_used_collection_shard1_replica_n4&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=201 2> 362337 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/nodes_used_collection/terms/shard2 to Terms{values={}, version=3} for removeTerm 2> 362337 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/nodes_used_collection/terms/shard1 to Terms{values={}, version=3} for removeTerm 2> 362339 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration. 2> 362339 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration. 2> 362341 INFO (qtp1052031273-2534) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=nodes_used_collection_shard1_replica_n6&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=205 2> 362341 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=nodes_used_collection_shard2_replica_n2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=203 2> 362447 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1364-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1352-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1328-thread-7) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1352-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1364-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1364-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1352-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1352-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1364-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362447 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/nodes_used_collection/state.json] for collection [nodes_used_collection] has occurred - updating... (live nodes size: [4]) 2> 362489 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=nodes_used_collection&action=DELETE&wt=javabin&version=2} status=0 QTime=364 2> 362492 INFO (TEST-CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending testCoresAreDistributedAcrossNodes 2> 362771 INFO (TEST-CollectionsAPIDistributedZkTest.testCollectionsAPI-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting testCollectionsAPI 2> 362787 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={async=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd&collection.configName=conf2&name=awhollynewcollection_0&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=7 2> 362792 INFO (TEST-CollectionsAPIDistributedZkTest.testCollectionsAPI-seed#[DF064544E215B325]) [] o.a.s.c.MiniSolrCloudCluster waitForActiveCollection: awhollynewcollection_0 2> 362794 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection awhollynewcollection_0 2> 362794 INFO (TEST-CollectionsAPIDistributedZkTest.testCollectionsAPI-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader /collections/awhollynewcollection_0/state.json is deleted, stop watching children 2> 362794 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000069 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 362905 INFO (zkCallback-1373-thread-13) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 363013 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"awhollynewcollection_0_shard1_replica_n1", 2> "node_name":"127.0.0.1:37221_solr", 2> "base_url":"http://127.0.0.1:37221/solr", 2> "collection":"awhollynewcollection_0", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 363018 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"awhollynewcollection_0_shard2_replica_n2", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "collection":"awhollynewcollection_0", 2> "shard":"shard2", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 363023 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"awhollynewcollection_0_shard2_replica_n4", 2> "node_name":"127.0.0.1:33871_solr", 2> "base_url":"http://127.0.0.1:33871/solr", 2> "collection":"awhollynewcollection_0", 2> "shard":"shard2", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 363026 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"awhollynewcollection_0_shard1_replica_n6", 2> "node_name":"127.0.0.1:40837_solr", 2> "base_url":"http://127.0.0.1:40837/solr", 2> "collection":"awhollynewcollection_0", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 363132 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 363132 INFO (zkCallback-1373-thread-13) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 363144 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr x:awhollynewcollection_0_shard1_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf2&newCollection=true&collection=awhollynewcollection_0&version=2&replicaType=NRT&async=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898&coreNodeName=core_node8&name=awhollynewcollection_0_shard1_replica_n6&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=0 QTime=4 2> 363144 INFO (qtp1052031273-2522) [n:127.0.0.1:37221_solr x:awhollynewcollection_0_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf2&newCollection=true&collection=awhollynewcollection_0&version=2&replicaType=NRT&async=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227&coreNodeName=core_node3&name=awhollynewcollection_0_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin} status=0 QTime=5 2> 363144 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr x:awhollynewcollection_0_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf2&newCollection=true&collection=awhollynewcollection_0&version=2&replicaType=NRT&async=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002&coreNodeName=core_node5&name=awhollynewcollection_0_shard2_replica_n2&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=0 QTime=5 2> 363145 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr x:awhollynewcollection_0_shard2_replica_n4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf2&newCollection=true&collection=awhollynewcollection_0&version=2&replicaType=NRT&async=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525&coreNodeName=core_node7&name=awhollynewcollection_0_shard2_replica_n4&action=CREATE&numShards=2&shard=shard2&wt=javabin} status=0 QTime=6 2> 363146 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr x:awhollynewcollection_0_shard1_replica_n6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf2&newCollection=true&collection=awhollynewcollection_0&version=2&replicaType=NRT&async=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898&coreNodeName=core_node8&name=awhollynewcollection_0_shard1_replica_n6&action=CREATE&numShards=2&shard=shard1&wt=javabin 2> 363146 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr x:awhollynewcollection_0_shard2_replica_n4] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf2&newCollection=true&collection=awhollynewcollection_0&version=2&replicaType=NRT&async=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525&coreNodeName=core_node7&name=awhollynewcollection_0_shard2_replica_n4&action=CREATE&numShards=2&shard=shard2&wt=javabin 2> 363146 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr x:awhollynewcollection_0_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf2&newCollection=true&collection=awhollynewcollection_0&version=2&replicaType=NRT&async=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227&coreNodeName=core_node3&name=awhollynewcollection_0_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&wt=javabin 2> 363146 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr x:awhollynewcollection_0_shard2_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf2&newCollection=true&collection=awhollynewcollection_0&version=2&replicaType=NRT&async=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002&coreNodeName=core_node5&name=awhollynewcollection_0_shard2_replica_n2&action=CREATE&numShards=2&shard=shard2&wt=javabin 2> 363148 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr] o.a.s.h.a.CoreAdminOperation Checking request status for : ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 2> 363148 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0 2> 363167 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 363170 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.s.IndexSchema Schema name=minimal 2> 363171 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 363172 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 363172 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'awhollynewcollection_0_shard2_replica_n2' using configuration from configset conf2, trusted=true 2> 363172 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 363173 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 363174 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/awhollynewcollection_0_shard2_replica_n2], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/awhollynewcollection_0_shard2_replica_n2/data/] 2> 363176 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.s.IndexSchema Schema name=minimal 2> 363177 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 363177 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.s.IndexSchema Schema name=minimal 2> 363177 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'awhollynewcollection_0_shard2_replica_n4' using configuration from configset conf2, trusted=true 2> 363178 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/awhollynewcollection_0_shard2_replica_n4], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node1/awhollynewcollection_0_shard2_replica_n4/data/] 2> 363178 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 363178 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'awhollynewcollection_0_shard1_replica_n6' using configuration from configset conf2, trusted=true 2> 363189 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node2/awhollynewcollection_0_shard1_replica_n6], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node2/awhollynewcollection_0_shard1_replica_n6/data/] 2> 363191 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.s.IndexSchema Schema name=minimal 2> 363193 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 363194 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'awhollynewcollection_0_shard1_replica_n1' using configuration from configset conf2, trusted=true 2> 363196 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node3/awhollynewcollection_0_shard1_replica_n1], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node3/awhollynewcollection_0_shard1_replica_n1/data/] 2> 363278 INFO (zkCallback-1373-thread-13) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 363349 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2> 363355 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 363355 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 363359 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 363360 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 363367 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf2 2> 363368 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf2 2> 363370 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 363370 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936578347073536 2> 363392 INFO (searcherExecutor-1586-thread-1-processing-awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 core_node5 create 127.0.0.1:33017_solr awhollynewcollection_0 shard2) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 363404 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/awhollynewcollection_0/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm 2> 363405 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/awhollynewcollection_0/leaders/shard2 2> 363414 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2> 363414 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2> 363419 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoute in=14998ms 2> 363423 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 363423 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 363423 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 363424 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 363487 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 363488 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 363488 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 363488 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 363489 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2> 363499 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 363499 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 363501 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf2 2> 363501 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf2 2> 363502 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf2 2> 363502 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf2 2> 363505 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 363505 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 363506 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 363506 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936578489679872 2> 363506 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936578489679872 2> 363506 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 363525 INFO (searcherExecutor-1590-thread-1-processing-awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 core_node8 create 127.0.0.1:40837_solr awhollynewcollection_0 shard1) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 363526 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf2 2> 363526 INFO (searcherExecutor-1588-thread-1-processing-awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 core_node7 create 127.0.0.1:33871_solr awhollynewcollection_0 shard2) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 363527 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf2 2> 363527 INFO (zkCallback-1373-thread-13) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 363528 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 363529 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936578513797120 2> 363550 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.c.ZkShardTerms Successful update of terms at /collections/awhollynewcollection_0/terms/shard2 to Terms{values={core_node7=0, core_node5=0}, version=1} for registerTerm 2> 363550 INFO (searcherExecutor-1592-thread-1-processing-awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 core_node3 create 127.0.0.1:37221_solr awhollynewcollection_0 shard1) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 363551 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/awhollynewcollection_0/terms/shard1 to Terms{values={core_node8=0}, version=0} for registerTerm 2> 363551 INFO (parallelCoreAdminAPIBaseExecutor-1344-thread-2-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 create) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/awhollynewcollection_0/leaders/shard2 2> 363551 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/awhollynewcollection_0/leaders/shard1 2> 363556 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/awhollynewcollection_0/terms/shard1 to Terms{values={core_node3=0, core_node8=0}, version=1} for registerTerm 2> 363556 INFO (parallelCoreAdminAPIBaseExecutor-1368-thread-2-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 create) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/awhollynewcollection_0/leaders/shard1 2> 363562 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 363563 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 363563 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40837/solr/awhollynewcollection_0_shard1_replica_n6/ 2> 363565 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.u.PeerSync PeerSync: core=awhollynewcollection_0_shard1_replica_n6 url=http://127.0.0.1:40837/solr START replicas=[http://127.0.0.1:37221/solr/awhollynewcollection_0_shard1_replica_n1/] nUpdates=100 2> 363568 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.u.PeerSync PeerSync: core=awhollynewcollection_0_shard1_replica_n6 url=http://127.0.0.1:40837/solr DONE. We have no versions. sync failed. 2> 363571 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.c.S.Request webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=1 2> 363573 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate 2> 363573 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway 2> 363573 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/awhollynewcollection_0/leaders/shard1/leader after winning as /collections/awhollynewcollection_0/leader_elect/shard1/election/72085948270444553-core_node8-n_0000000000 2> 363587 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40837/solr/awhollynewcollection_0_shard1_replica_n6/ shard1 2> 363692 INFO (zkCallback-1373-thread-13) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 363692 INFO (zkCallback-1352-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 363692 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 363696 INFO (parallelCoreAdminAPIBaseExecutor-1357-thread-2-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 create) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 363807 INFO (zkCallback-1352-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 363807 INFO (zkCallback-1352-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 363807 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 363807 INFO (zkCallback-1373-thread-13) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 363923 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 363924 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 363924 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33017/solr/awhollynewcollection_0_shard2_replica_n2/ 2> 363926 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.u.PeerSync PeerSync: core=awhollynewcollection_0_shard2_replica_n2 url=http://127.0.0.1:33017/solr START replicas=[http://127.0.0.1:33871/solr/awhollynewcollection_0_shard2_replica_n4/] nUpdates=100 2> 363928 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.u.PeerSync PeerSync: core=awhollynewcollection_0_shard2_replica_n2 url=http://127.0.0.1:33017/solr DONE. We have no versions. sync failed. 2> 363933 INFO (qtp699896367-2519) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.c.S.Request webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=2 2> 363935 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate 2> 363935 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway 2> 363935 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/awhollynewcollection_0/leaders/shard2/leader after winning as /collections/awhollynewcollection_0/leader_elect/shard2/election/72085948270444552-core_node5-n_0000000000 2> 363943 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33017/solr/awhollynewcollection_0_shard2_replica_n2/ shard2 2> 364049 INFO (zkCallback-1352-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364049 INFO (zkCallback-1373-thread-13) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364049 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364049 INFO (zkCallback-1352-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364049 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364052 INFO (parallelCoreAdminAPIBaseExecutor-1346-thread-2-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 create) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 364155 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr] o.a.s.h.a.CoreAdminOperation Checking request status for : ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 2> 364155 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=1 2> 364161 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364161 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364161 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364161 INFO (zkCallback-1352-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364161 INFO (zkCallback-1352-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364161 INFO (zkCallback-1373-thread-13) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364688 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364688 INFO (zkCallback-1352-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364688 INFO (zkCallback-1373-thread-13) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364688 INFO (zkCallback-1352-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364688 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364688 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364688 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364688 INFO (zkCallback-1328-thread-7) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364688 INFO (zkCallback-1364-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364688 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 364694 INFO (TEST-CollectionsAPIDistributedZkTest.testCollectionsAPI-seed#[DF064544E215B325]) [] o.a.s.c.SolrCloudTestCase waitForState (awhollynewcollection_0): Expected to see collection awhollynewcollection_0 2> 365163 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr] o.a.s.h.a.CoreAdminOperation Checking request status for : ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227 2> 365164 INFO (qtp1052031273-2542) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708048100227&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=1 2> 365168 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.h.a.CoreAdminOperation Checking request status for : ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002 2> 365168 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708052674002&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=1 2> 365171 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.h.a.CoreAdminOperation Checking request status for : ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525 2> 365171 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708057395525&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0 2> 365174 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr] o.a.s.h.a.CoreAdminOperation Checking request status for : ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898 2> 365174 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=ab51a331-5fb2-4fec-93d3-bbbb0e7991bd432708061402898&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=1 2> 365182 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000071 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 365802 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/awhollynewcollection_0/terms/shard1 to Terms{values={core_node3=1, core_node8=1}, version=2} for ensureHighestTermsAreNotZero 2> 365803 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/awhollynewcollection_0/terms/shard2 to Terms{values={core_node7=1, core_node5=1}, version=2} for ensureHighestTermsAreNotZero 2> 365849 INFO (qtp1052031273-2536) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:40837/solr/awhollynewcollection_0_shard1_replica_n6/&wt=javabin&version=2}{add=[8 (1775936580837441536)]} 0 29 2> 365850 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:33017/solr/awhollynewcollection_0_shard2_replica_n2/&wt=javabin&version=2}{add=[6 (1775936580837441536), 7 (1775936580880433152)]} 0 32 2> 365856 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={_stateVer_=awhollynewcollection_0:8&wt=javabin&version=2}{add=[6 (1775936580837441536), 7 (1775936580880433152)]} 0 131 2> 365856 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={_stateVer_=awhollynewcollection_0:8&wt=javabin&version=2}{add=[8 (1775936580837441536)]} 0 131 2> 366047 INFO (searcherExecutor-1590-thread-1-processing-127.0.0.1:40837_solr awhollynewcollection_0_shard1_replica_n6 awhollynewcollection_0 shard1 core_node8) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.SolrCore Registered new searcher autowarm time: 1 ms 2> 366050 INFO (searcherExecutor-1592-thread-1-processing-127.0.0.1:37221_solr awhollynewcollection_0_shard1_replica_n1 awhollynewcollection_0 shard1 core_node3) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.c.SolrCore Registered new searcher autowarm time: 1 ms 2> 366051 INFO (qtp1052031273-2528) [n:127.0.0.1:37221_solr c:awhollynewcollection_0 s:shard1 r:core_node3 x:awhollynewcollection_0_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:40837/solr/awhollynewcollection_0_shard1_replica_n6/&commit_end_point=replicas&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 156 2> 366056 INFO (qtp772667146-2526) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={update.distrib=TOLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:33017/solr/awhollynewcollection_0_shard2_replica_n2/&commit_end_point=leaders&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 177 2> 366142 INFO (searcherExecutor-1586-thread-1-processing-127.0.0.1:33017_solr awhollynewcollection_0_shard2_replica_n2 awhollynewcollection_0 shard2 core_node5) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 366148 INFO (searcherExecutor-1588-thread-1-processing-127.0.0.1:33871_solr awhollynewcollection_0_shard2_replica_n4 awhollynewcollection_0 shard2 core_node7) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 366149 INFO (qtp699896367-2535) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:33017/solr/awhollynewcollection_0_shard2_replica_n2/&commit_end_point=replicas&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 73 2> 366152 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr c:awhollynewcollection_0 s:shard2 r:core_node5 x:awhollynewcollection_0_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={_stateVer_=awhollynewcollection_0:8&commit=true&wt=javabin&version=2}{commit=} 0 286 2> 366327 INFO (qtp699896367-2531) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.c.S.Request webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=awhollynewcollection_0:8&fl=id&fl=score&shards.purpose=16388&start=0&fsv=true&shard.url=http://127.0.0.1:33871/solr/awhollynewcollection_0_shard2_replica_n4/|http://127.0.0.1:33017/solr/awhollynewcollection_0_shard2_replica_n2/&rid=127.0.0.1-3704&rows=10&version=2&q=*:*&omitHeader=false&NOW=1693665105129&isShard=true&wt=javabin} hits=2 status=0 QTime=42 2> 366327 INFO (qtp772667146-2524) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.S.Request webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=awhollynewcollection_0:8&fl=id&fl=score&shards.purpose=16388&start=0&fsv=true&shard.url=http://127.0.0.1:40837/solr/awhollynewcollection_0_shard1_replica_n6/|http://127.0.0.1:37221/solr/awhollynewcollection_0_shard1_replica_n1/&rid=127.0.0.1-3704&rows=10&version=2&q=*:*&omitHeader=false&NOW=1693665105129&isShard=true&wt=javabin} hits=1 status=0 QTime=42 2> 366473 INFO (qtp772667146-2532) [n:127.0.0.1:40837_solr c:awhollynewcollection_0 s:shard1 r:core_node8 x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.S.Request webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=awhollynewcollection_0:8&shards.purpose=64&shard.url=http://127.0.0.1:40837/solr/awhollynewcollection_0_shard1_replica_n6/|http://127.0.0.1:37221/solr/awhollynewcollection_0_shard1_replica_n1/&rid=127.0.0.1-3704&version=2&q=*:*&omitHeader=false&NOW=1693665105129&ids=8&isShard=true&wt=javabin} status=0 QTime=18 2> 366473 INFO (qtp699896367-2537) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.c.S.Request webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=awhollynewcollection_0:8&shards.purpose=64&shard.url=http://127.0.0.1:33871/solr/awhollynewcollection_0_shard2_replica_n4/|http://127.0.0.1:33017/solr/awhollynewcollection_0_shard2_replica_n2/&rid=127.0.0.1-3704&version=2&q=*:*&omitHeader=false&NOW=1693665105129&ids=6,7&isShard=true&wt=javabin} status=0 QTime=18 2> 366479 INFO (qtp699896367-2525) [n:127.0.0.1:33871_solr c:awhollynewcollection_0 s:shard2 r:core_node7 x:awhollynewcollection_0_shard2_replica_n4] o.a.s.c.S.Request webapp=/solr path=/select params={q=*:*&_stateVer_=awhollynewcollection_0:8&wt=javabin&version=2} rid=127.0.0.1-3704 hits=3 status=0 QTime=317 2> 366825 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000071 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 366827 INFO (TEST-CollectionsAPIDistributedZkTest.testCollectionsAPI-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 366897 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 366904 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr x:awhollynewcollection_0_shard2_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.awhollynewcollection_0.shard2.replica_n4 tag=null 2> 366904 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:awhollynewcollection_0_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.awhollynewcollection_0.shard2.replica_n2 tag=null 2> 366904 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr x:awhollynewcollection_0_shard1_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.awhollynewcollection_0.shard1.replica_n6 tag=null 2> 366904 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr x:awhollynewcollection_0_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.awhollynewcollection_0.shard1.replica_n1 tag=null 2> 366998 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr x:awhollynewcollection_0_shard2_replica_n4] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@5c0e0702 awhollynewcollection_0_shard2_replica_n4 2> 367000 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr x:awhollynewcollection_0_shard2_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.awhollynewcollection_0.shard2.replica_n4 tag=SolrCore@5c0e0702 2> 367000 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr x:awhollynewcollection_0_shard2_replica_n4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.awhollynewcollection_0.shard2.leader tag=SolrCore@5c0e0702 2> 367020 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr x:awhollynewcollection_0_shard2_replica_n4] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 367046 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:awhollynewcollection_0_shard2_replica_n2] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@1046e30a awhollynewcollection_0_shard2_replica_n2 2> 367047 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:awhollynewcollection_0_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.awhollynewcollection_0.shard2.replica_n2 tag=SolrCore@1046e30a 2> 367048 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:awhollynewcollection_0_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.awhollynewcollection_0.shard2.leader tag=SolrCore@1046e30a 2> 367060 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:awhollynewcollection_0_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 367077 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr x:awhollynewcollection_0_shard1_replica_n1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@39367734 awhollynewcollection_0_shard1_replica_n1 2> 367077 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr x:awhollynewcollection_0_shard1_replica_n6] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@66e68d90 awhollynewcollection_0_shard1_replica_n6 2> 367078 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr x:awhollynewcollection_0_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.awhollynewcollection_0.shard1.replica_n1 tag=SolrCore@39367734 2> 367078 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr x:awhollynewcollection_0_shard1_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.awhollynewcollection_0.shard1.replica_n6 tag=SolrCore@66e68d90 2> 367078 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr x:awhollynewcollection_0_shard1_replica_n6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.awhollynewcollection_0.shard1.leader tag=SolrCore@66e68d90 2> 367078 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr x:awhollynewcollection_0_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.awhollynewcollection_0.shard1.leader tag=SolrCore@39367734 2> 367093 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr x:awhollynewcollection_0_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 367093 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr x:awhollynewcollection_0_shard1_replica_n6] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 367126 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr] o.a.s.c.ZkShardTerms Failed to save terms, version is not a match, retrying 2> 367126 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr] o.a.s.c.ZkShardTerms Failed to save terms, version is not a match, retrying 2> 367126 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/awhollynewcollection_0/terms/shard1 to Terms{values={core_node3=1}, version=3} for removeTerm 2> 367126 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/awhollynewcollection_0/terms/shard2 to Terms{values={core_node7=1}, version=3} for removeTerm 2> 367130 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/awhollynewcollection_0/terms/shard1 to Terms{values={}, version=4} for removeTerm 2> 367131 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/awhollynewcollection_0/terms/shard2 to Terms{values={}, version=4} for removeTerm 2> 367137 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration. 2> 367138 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration. 2> 367139 WARN (zkCallback-1364-thread-2) [] o.a.s.c.LeaderElector Our node is no longer in line to be leader 2> 367139 WARN (zkCallback-1328-thread-1) [] o.a.s.c.LeaderElector Our node is no longer in line to be leader 2> 367151 INFO (qtp699896367-2539) [n:127.0.0.1:33871_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=awhollynewcollection_0_shard2_replica_n4&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=248 2> 367151 INFO (qtp1052031273-2538) [n:127.0.0.1:37221_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=awhollynewcollection_0_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=250 2> 367153 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=awhollynewcollection_0_shard2_replica_n2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=250 2> 367153 INFO (qtp772667146-2518) [n:127.0.0.1:40837_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=awhollynewcollection_0_shard1_replica_n6&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=252 2> 367163 INFO (zkCallback-1364-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367164 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367163 INFO (zkCallback-1352-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367163 INFO (zkCallback-1352-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367163 INFO (zkCallback-1364-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367164 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367164 INFO (zkCallback-1328-thread-7) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367164 INFO (zkCallback-1352-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367164 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367276 INFO (zkCallback-1328-thread-7) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367276 INFO (zkCallback-1328-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367276 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367276 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367276 INFO (zkCallback-1328-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367276 INFO (zkCallback-1328-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/awhollynewcollection_0/state.json] for collection [awhollynewcollection_0] has occurred - updating... (live nodes size: [4]) 2> 367300 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=awhollynewcollection_0&action=DELETE&wt=javabin&version=2} status=0 QTime=423 2> 367303 INFO (TEST-CollectionsAPIDistributedZkTest.testCollectionsAPI-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending testCollectionsAPI 2> 367658 INFO (TEST-CollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Starting testCreationAndDeletion 2> 367675 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CreateCollectionCmd Create collection created_and_deleted 2> 367676 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000072 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 367900 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.o.SliceMutator createReplica() { 2> "core":"created_and_deleted_shard1_replica_n1", 2> "node_name":"127.0.0.1:33017_solr", 2> "base_url":"http://127.0.0.1:33017/solr", 2> "collection":"created_and_deleted", 2> "shard":"shard1", 2> "state":"down", 2> "type":"NRT", 2> "operation":"ADDREPLICA", 2> "waitForFinalState":"false"} 2> 368007 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4]) 2> 368015 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr x:created_and_deleted_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=created_and_deleted_shard1_replica_n1&action=CREATE&numShards=1&collection=created_and_deleted&shard=shard1&wt=javabin&version=2&replicaType=NRT 2> 368055 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.4.2 2> 368065 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.s.IndexSchema Schema name=minimal 2> 368071 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id 2> 368072 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'created_and_deleted_shard1_replica_n1' using configuration from configset conf, trusted=true 2> 368079 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.SolrCore Opening new SolrCore at [/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/created_and_deleted_shard1_replica_n1], dataDir=[/build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001/tempDir-001/node4/created_and_deleted_shard1_replica_n1/data/] 2> 368229 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2> 368229 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 2> 368236 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled 2> 368236 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2> 368259 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 2> 368262 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 2> 368265 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2> 368267 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1775936583481950208 2> 368289 INFO (searcherExecutor-1612-thread-1-processing-127.0.0.1:33017_solr created_and_deleted_shard1_replica_n1 created_and_deleted shard1 core_node2) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2> 368301 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/created_and_deleted/terms/shard1 to Terms{values={core_node2=0}, version=0} for registerTerm 2> 368302 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/created_and_deleted/leaders/shard1 2> 368313 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2> 368315 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2> 368315 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33017/solr/created_and_deleted_shard1_replica_n1/ 2> 368317 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2> 368318 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:33017/solr/created_and_deleted_shard1_replica_n1/ has no replicas 2> 368318 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/created_and_deleted/leaders/shard1/leader after winning as /collections/created_and_deleted/leader_elect/shard1/election/72085948270444552-core_node2-n_0000000000 2> 368326 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33017/solr/created_and_deleted_shard1_replica_n1/ shard1 2> 368432 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4]) 2> 368432 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4]) 2> 368436 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr c:created_and_deleted s:shard1 r:core_node2 x:created_and_deleted_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary 2> 368449 INFO (qtp1260674627-2533) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=created_and_deleted_shard1_replica_n1&action=CREATE&numShards=1&collection=created_and_deleted&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=435 2> 368459 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2> 368546 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4]) 2> 368546 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4]) 2> 368546 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4]) 2> 368548 INFO (qtp1260674627-2517) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf&name=created_and_deleted&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=882 2> 368556 INFO (qtp1260674627-2523) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=0 2> 368566 INFO (OverseerCollectionConfigSetProcessor-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000074 doesn't exist. Requestor may have disconnected from ZooKeeper 2> 368568 INFO (OverseerThreadFactory-1341-thread-5) [] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 2> 368572 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:created_and_deleted_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.created_and_deleted.shard1.replica_n1 tag=null 2> 368679 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:created_and_deleted_shard1_replica_n1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@2fd83f1d created_and_deleted_shard1_replica_n1 2> 368680 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:created_and_deleted_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.created_and_deleted.shard1.replica_n1 tag=SolrCore@2fd83f1d 2> 368680 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:created_and_deleted_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.created_and_deleted.shard1.leader tag=SolrCore@2fd83f1d 2> 368682 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr x:created_and_deleted_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 2> 368706 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/created_and_deleted/terms/shard1 to Terms{values={}, version=1} for removeTerm 2> 368710 INFO (qtp1260674627-2529) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=created_and_deleted_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=139 2> 368820 INFO (zkCallback-1337-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4]) 2> 368820 INFO (zkCallback-1337-thread-3) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4]) 2> 368820 INFO (zkCallback-1328-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4]) 2> 368820 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4]) 2> 368820 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4]) 2> 368820 INFO (zkCallback-1337-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4]) 2> 368820 INFO (zkCallback-1337-thread-6) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4]) 2> 368835 INFO (qtp1260674627-2521) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=created_and_deleted&action=DELETE&wt=javabin&version=2} status=0 QTime=276 2> 368841 INFO (qtp1260674627-2527) [n:127.0.0.1:33017_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=0 2> 368846 INFO (TEST-CollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[DF064544E215B325]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) 2> 368850 INFO (TEST-CollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[DF064544E215B325]) [] o.a.s.SolrTestCaseJ4 ###Ending testCreationAndDeletion 2> 368973 INFO (jetty-closer-1619-thread-2) [] o.e.j.s.Server Stopped Server@35901618{STOPPING}[10.0.13,sto=0] 2> 368973 INFO (jetty-closer-1619-thread-3) [] o.e.j.s.Server Stopped Server@50dcf77c{STOPPING}[10.0.13,sto=0] 2> 368973 INFO (jetty-closer-1619-thread-1) [] o.e.j.s.Server Stopped Server@57e1a769{STOPPING}[10.0.13,sto=0] 2> 368973 INFO (jetty-closer-1619-thread-4) [] o.e.j.s.Server Stopped Server@40f59aa7{STOPPING}[10.0.13,sto=0] 2> 368991 INFO (jetty-closer-1619-thread-1) [] o.e.j.s.AbstractConnector Stopped ServerConnector@27706969{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0} 2> 368991 INFO (jetty-closer-1619-thread-3) [] o.e.j.s.AbstractConnector Stopped ServerConnector@20ca1cbb{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0} 2> 368991 INFO (jetty-closer-1619-thread-2) [] o.e.j.s.AbstractConnector Stopped ServerConnector@45e3b957{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0} 2> 368991 INFO (jetty-closer-1619-thread-4) [] o.e.j.s.AbstractConnector Stopped ServerConnector@3073408f{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0} 2> 368996 INFO (jetty-closer-1619-thread-1) [] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4dca3202{/solr,file:///build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-cwd/,STOPPED} 2> 368996 INFO (jetty-closer-1619-thread-3) [] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@780b670c{/solr,file:///build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-cwd/,STOPPED} 2> 368997 INFO (jetty-closer-1619-thread-4) [] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6c11fe{/solr,file:///build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-cwd/,STOPPED} 2> 368997 INFO (jetty-closer-1619-thread-2) [] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@c8983c4{/solr,file:///build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-cwd/,STOPPED} 2> 369009 INFO (jetty-closer-1619-thread-4) [] o.a.s.c.CoreContainer Shutting down CoreContainer instance=733124444 2> 369009 INFO (jetty-closer-1619-thread-3) [] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2094329570 2> 369009 INFO (jetty-closer-1619-thread-1) [] o.a.s.c.CoreContainer Shutting down CoreContainer instance=479648059 2> 369010 INFO (jetty-closer-1619-thread-1) [] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:33871_solr 2> 369010 INFO (jetty-closer-1619-thread-3) [] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:40837_solr 2> 369010 INFO (jetty-closer-1619-thread-4) [] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:37221_solr 2> 369016 INFO (jetty-closer-1619-thread-3) [] o.a.s.c.ZkController Publish this node as DOWN... 2> 369016 INFO (jetty-closer-1619-thread-4) [] o.a.s.c.ZkController Publish this node as DOWN... 2> 369016 INFO (jetty-closer-1619-thread-3) [] o.a.s.c.ZkController Publish node=127.0.0.1:40837_solr as DOWN 2> 369016 INFO (jetty-closer-1619-thread-4) [] o.a.s.c.ZkController Publish node=127.0.0.1:37221_solr as DOWN 2> 369016 INFO (jetty-closer-1619-thread-1) [] o.a.s.c.ZkController Publish this node as DOWN... 2> 369016 INFO (jetty-closer-1619-thread-1) [] o.a.s.c.ZkController Publish node=127.0.0.1:33871_solr as DOWN 2> 369018 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (1) 2> 369018 INFO (zkCallback-1364-thread-4) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (1) 2> 369018 INFO (zkCallback-1352-thread-4) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (1) 2> 369018 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (1) 2> 369032 INFO (jetty-closer-1619-thread-1) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null 2> 369032 INFO (jetty-closer-1619-thread-4) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null 2> 369032 INFO (jetty-closer-1619-thread-3) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null 2> 369058 INFO (jetty-closer-1619-thread-3) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null 2> 369062 INFO (jetty-closer-1619-thread-1) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null 2> 369071 INFO (jetty-closer-1619-thread-2) [] o.a.s.c.CoreContainer Shutting down CoreContainer instance=875191349 2> 369071 INFO (jetty-closer-1619-thread-2) [] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:33017_solr 2> 369072 INFO (jetty-closer-1619-thread-3) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null 2> 369072 INFO (jetty-closer-1619-thread-3) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null 2> 369074 INFO (jetty-closer-1619-thread-2) [] o.a.s.c.ZkController Publish this node as DOWN... 2> 369074 INFO (jetty-closer-1619-thread-2) [] o.a.s.c.ZkController Publish node=127.0.0.1:33017_solr as DOWN 2> 369080 INFO (jetty-closer-1619-thread-1) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null 2> 369081 INFO (zkCallback-1352-thread-4) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0) 2> 369081 INFO (jetty-closer-1619-thread-1) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null 2> 369081 INFO (zkCallback-1337-thread-4) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0) 2> 369081 INFO (zkCallback-1328-thread-5) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0) 2> 369081 INFO (zkCallback-1364-thread-4) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0) 2> 369083 INFO (jetty-closer-1619-thread-2) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null 2> 369089 INFO (closeThreadPool-1629-thread-1) [] o.a.s.c.Overseer Overseer (id=72085948270444551-127.0.0.1:33871_solr-n_0000000000) closing 2> 369096 INFO (OverseerStateUpdate-72085948270444551-127.0.0.1:33871_solr-n_0000000000) [n:127.0.0.1:33871_solr] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:33871_solr 2> 369098 INFO (closeThreadPool-1629-thread-2) [] o.a.s.c.Overseer Overseer (id=72085948270444551-127.0.0.1:33871_solr-n_0000000000) closing 2> 369126 INFO (jetty-closer-1619-thread-4) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null 2> 369130 INFO (jetty-closer-1619-thread-2) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null 2> 369150 INFO (jetty-closer-1619-thread-2) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null 2> 369151 INFO (jetty-closer-1619-thread-2) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null 2> 369159 INFO (jetty-closer-1619-thread-4) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null 2> 369159 INFO (jetty-closer-1619-thread-4) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null 2> 369210 INFO (jetty-closer-1619-thread-1) [] o.a.s.c.Overseer Overseer (id=72085948270444551-127.0.0.1:33871_solr-n_0000000000) closing 2> 369267 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.c.ZkTestServer Shutting down ZkTestServer. 2> 369483 WARN (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer Watch limit violations: 2> Maximum concurrent create/delete watches above limit: 2> 2> 29 /solr/collections/addReplicaColl/terms/shard1 2> 22 /solr/aliases.json 2> 22 /solr/configs/conf 2> 22 /solr/clusterprops.json 2> 21 /solr/collections/nodes_used_collection/state.json 2> 12 /solr/collections/addReplicaColl/state.json 2> 11 /solr/collections/onlyinzk/state.json 2> 11 /solr/collections/acollectionafterbaddelete/state.json 2> 10 /solr/collections/halfdeletedcollection/state.json 2> 10 /solr/collections/awhollynewcollection_0/terms/shard1 2> 10 /solr/collections/awhollynewcollection_0/terms/shard2 2> 9 /solr/collections/addReplicaColl/terms/shard2 2> 8 /solr/collections/acollectionafterbaddelete/terms/shard1 2> 8 /solr/collections/nodes_used_collection/terms/shard2 2> 8 /solr/collections/reloaded_collection/terms/shard1 2> 8 /solr/collections/reloaded_collection/terms/shard2 2> 7 /solr/collections/awhollynewcollection_0/state.json 2> 7 /solr/collections/created_and_deleted/state.json 2> 7 /solr/collections/nodes_used_collection/terms/shard1 2> 7 /solr/collections/reloaded_collection/state.json 2> 7 /solr/collections/withconfigset2/state.json 2> 6 /solr/collections/halfcollectionblocker/state.json 2> 6 /solr/collections/halfdeletedcollection/terms/shard1 2> 5 /solr/configs/conf2 2> 4 /solr/collections/addReplicaColl/collectionprops.json 2> 4 /solr/packages.json 2> 4 /solr/security.json 2> 4 /solr/collections/awhollynewcollection_0/collectionprops.json 2> 3 /solr/collections/created_and_deleted/terms/shard1 2> 3 /solr/collections/halfcollectionblocker2/state.json 2> 3 /solr/collections/halfcollectionblocker2/terms/shard1 2> 3 /solr/collections/withconfigset2/terms/shard1 2> 3 /solr/collections/halfdeletedcollection/terms/shard2 2> 3 /solr/collections/halfcollectionblocker/terms/shard1 2> 3 /solr/collections/onlyinzk/terms/shard2 2> 3 /solr/collections/onlyinzk/terms/shard1 2> 2 /solr/collections/halfcollection/state.json 2> 2> Maximum concurrent data watches above limit: 2> 2> 238 /solr/aliases.json 2> 114 /solr/collections/addReplicaColl/state.json 2> 65 /solr/collections/awhollynewcollection_0/state.json 2> 63 /solr/collections/reloaded_collection/state.json 2> 49 /solr/collections/halfdeletedcollection/state.json 2> 45 /solr/collections/nodes_used_collection/state.json 2> 23 /solr/collections/acollectionafterbaddelete/state.json 2> 22 /solr/collections/onlyinzk/state.json 2> 18 /solr/collections/halfcollectionblocker2/state.json 2> 13 /solr/collections/halfcollectionblocker/state.json 2> 13 /solr/collections/created_and_deleted/state.json 2> 13 /solr/collections/withconfigset2/state.json 2> 3 /solr/collections/halfcollection/state.json 2> 2 /solr/collections/nodeset_collection/state.json 2> 2 /solr/overseer_elect/election/72085948270444552-127.0.0.1:33017_solr-n_0000000001 2> 2 /solr/collections/addReplicaColl/leader_elect/shard1/election/72085948270444554-core_node10-n_0000000002 2> 2 /solr/collections/addReplicaColl/leader_elect/shard1/election/72085948270444554-core_node5-n_0000000000 2> 2> Maximum concurrent children watches above limit: 2> 2> 248 /solr/overseer/queue 2> 192 /solr/collections 2> 131 /solr/overseer/collection-queue-work 2> 40 /solr/live_nodes 2> 23 /solr/collections/addReplicaColl/state.json 2> 23 /solr/collections/awhollynewcollection_0/state.json 2> 22 /solr/collections/reloaded_collection/state.json 2> 21 /solr/collections/nodes_used_collection/state.json 2> 21 /solr/collections/halfdeletedcollection/state.json 2> 12 /solr/collections/acollectionafterbaddelete/state.json 2> 11 /solr/collections/onlyinzk/state.json 2> 9 /solr/collections/halfcollectionblocker2/state.json 2> 7 /solr/collections/halfcollectionblocker/state.json 2> 7 /solr/collections/withconfigset2/state.json 2> 7 /solr/collections/created_and_deleted/state.json 2> 2 /solr/collections/nodeset_collection/state.json 2> 2 /solr/collections/halfcollection/state.json 2> 2> 369527 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-43 after mutting 0 log messages 2> 369527 INFO (SUITE-CollectionsAPIDistributedZkTest-seed#[DF064544E215B325]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-44 for ERROR logs matching regex: ignore_exception 2> NOTE: leaving temporary files on disk at: /build/solr/src/solr-9.2.1/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.CollectionsAPIDistributedZkTest_DF064544E215B325-001 2> NOTE: test params are: codec=Asserting(Lucene94): {_root_=PostingsFormat(name=LuceneVarGapFixedInterval), id=PostingsFormat(name=MockRandom)}, docValues:{}, maxPointsInLeafNode=33, maxMBSortInHeap=6.4712800327955975, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=fo-FO, timezone=America/Argentina/Buenos_Aires 2> NOTE: Linux 6.3.9-arch1-1 riscv64/Oracle Corporation 11.0.20 (64-bit)/cpus=1,threads=2,free=191179488,total=406847488 2> NOTE: All tests run in this JVM: [CursorPagingTest, TestCursorMarkWithoutUniqueKey, TestTolerantSearch, TestEmbeddedSolrServerAdminHandler, ChaosMonkeySafeLeaderTest, ConfigSetApiLockingTest, DistribJoinFromCollectionTest, LeaderElectionIntegrationTest, NodeMutatorTest, ParallelCommitExecutionTest, SSLMigrationTest, TestAuthenticationFramework, TestConfigSetsAPIExclusivity, TestLRUStatsCacheCloud, TestQueryingOnDownCollection, TestStressInPlaceUpdates, ZkControllerTest, CollectionsAPIDistributedZkTest] > Task :solr:solr-ref-guide:wipeTaskTemp > Task :solr:modules:hadoop-auth:compileTestJava > Task :solr:modules:clustering:wipeTaskTemp > Task :solr:test-framework:test :solr:test-framework:test (SUCCESS): 17 test(s) > Task :solr:modules:extraction:wipeTaskTemp > Task :solr:modules:hdfs:compileTestJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:modules:hdfs:testClasses > Task :solr:modules:jaegertracer-configurator:compileTestJava > Task :solr:modules:hdfs:test > Task :solr:core:test WARNING: Test org.apache.solr.TestRandomFaceting wrote 12,532,575 bytes of output. > Task :solr:solrj-zookeeper:test :solr:solrj-zookeeper:test (SUCCESS): 15 test(s) > Task :solr:prometheus-exporter:test :solr:prometheus-exporter:test (SUCCESS): 21 test(s) > Task :solr:prometheus-exporter:wipeTaskTemp > Task :solr:solrj-zookeeper:wipeTaskTemp > Task :solr:test-framework:wipeTaskTemp > Task :solr:modules:ltr:classes UP-TO-DATE > Task :solr:modules:ltr:compileTestJava Note: /build/solr/src/solr-9.2.1/solr/modules/ltr/src/test/org/apache/solr/ltr/TestLTROnSolrCloud.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:modules:jwt-auth:compileTestJava > Task :solr:modules:gcs-repository:test :solr:modules:gcs-repository:test (SUCCESS): 14 test(s), 1 skipped > Task :solr:modules:langid:compileTestJava > Task :solr:solrj-streaming:test :solr:solrj-streaming:test (SUCCESS): 604 test(s), 8 skipped > Task :solr:modules:langid:testClasses > Task :solr:modules:analysis-extras:test :solr:modules:analysis-extras:test (SUCCESS): 31 test(s) > Task :solr:modules:langid:test > Task :solr:solrj:test :solr:solrj:test (SUCCESS): 1088 test(s), 11 skipped > Task :solr:modules:gcs-repository:wipeTaskTemp > Task :solr:modules:opentelemetry:classes UP-TO-DATE > Task :solr:modules:analysis-extras:wipeTaskTemp > Task :solr:modules:s3-repository:classes UP-TO-DATE > Task :solr:solrj-streaming:wipeTaskTemp > Task :solr:modules:scripting:classes UP-TO-DATE > Task :solr:modules:scripting:compileTestJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:solrj:wipeTaskTemp > Task :solr:modules:analytics:test :solr:modules:analytics:test (SUCCESS): 472 test(s) > Task :solr:modules:opentelemetry:compileTestJava > Task :solr:modules:opentelemetry:testClasses > Task :solr:modules:sql:compileTestJava > Task :solr:modules:sql:testClasses > Task :solr:modules:s3-repository:compileTestJava > Task :solr:modules:langid:test :solr:modules:langid:test (SUCCESS): 43 test(s) > Task :solr:modules:langid:wipeTaskTemp > Task :solr:modules:jwt-auth:compileTestJava Note: /build/solr/src/solr-9.2.1/solr/modules/jwt-auth/src/test/org/apache/solr/security/jwt/JWTAuthPluginIntegrationTest.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:modules:jwt-auth:testClasses > Task :solr:modules:opentelemetry:test > Task :solr:modules:analytics:wipeTaskTemp > Task :solr:modules:sql:test > Task :solr:modules:ltr:testClasses > Task :solr:modules:jwt-auth:test > Task :solr:modules:ltr:test > Task :solr:modules:hadoop-auth:compileTestJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :solr:modules:s3-repository:testClasses > Task :solr:modules:scripting:testClasses > Task :solr:modules:jaegertracer-configurator:testClasses > Task :solr:modules:sql:test :solr:modules:sql:test (SUCCESS): 33 test(s) > Task :solr:modules:opentelemetry:test :solr:modules:opentelemetry:test (SUCCESS): 5 test(s) > Task :solr:modules:opentelemetry:wipeTaskTemp > Task :solr:modules:sql:wipeTaskTemp > Task :solr:modules:s3-repository:test > Task :solr:modules:scripting:test > Task :solr:modules:hdfs:test :solr:modules:hdfs:test (SUCCESS): 101 test(s), 43 skipped > Task :solr:modules:hadoop-auth:testClasses > Task :solr:modules:jaegertracer-configurator:test > Task :solr:modules:scripting:test :solr:modules:scripting:test (SUCCESS): 22 test(s), 1 skipped > Task :solr:modules:scripting:wipeTaskTemp > Task :solr:modules:hdfs:wipeTaskTemp > Task :solr:modules:ltr:test :solr:modules:ltr:test (SUCCESS): 169 test(s) > Task :solr:modules:jwt-auth:test :solr:modules:jwt-auth:test (SUCCESS): 61 test(s) > Task :solr:modules:jwt-auth:wipeTaskTemp > Task :solr:modules:hadoop-auth:test > Task :solr:modules:jaegertracer-configurator:test :solr:modules:jaegertracer-configurator:test (SUCCESS): 1 test(s) > Task :solr:modules:jaegertracer-configurator:wipeTaskTemp > Task :solr:modules:ltr:wipeTaskTemp > Task :solr:modules:hadoop-auth:test :solr:modules:hadoop-auth:test (SUCCESS): 55 test(s), 1 skipped > Task :solr:modules:hadoop-auth:wipeTaskTemp > Task :solr:modules:s3-repository:test :solr:modules:s3-repository:test (SUCCESS): 42 test(s), 1 skipped > Task :solr:modules:s3-repository:wipeTaskTemp > Task :solr:core:test WARNING: Test org.apache.solr.schema.TestCloudSchemaless wrote 11,855,324 bytes of output. :solr:core:test (FAILURE): 4651 test(s), 1 failure(s), 213 skipped 4651 tests completed, 1 failed, 213 skipped > Task :solr:core:test FAILED > Task :solr:core:wipeTaskTemp ERROR: The following test(s) have failed: - org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection (:solr:core) Test output: /build/solr/src/solr-9.2.1/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.txt Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection" -Ptests.jvms=12 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=DF064544E215B325 -Ptests.file.encoding=US-ASCII FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':solr:core:test'. > There were failing tests. See the results at: file:///build/solr/src/solr-9.2.1/solr/core/build/test-results/test/ * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. * Get more help at https://help.gradle.org BUILD FAILED in 34m 38s 138 actionable tasks: 102 executed, 36 up-to-date ==> ERROR: A failure occurred in check().  Aborting... ==> ERROR: Build failed, check /var/lib/archbuild/extra-riscv64/root23/build receiving incremental file list solr-9.2.1-1-riscv64-build.log solr-9.2.1-1-riscv64-check.log solr-9.2.1-1-riscv64-prepare.log sent 81 bytes received 55,700 bytes 37,187.33 bytes/sec total size is 674,236 speedup is 12.09