Bug 27165 - WildFly 20 won't start with our Java 11 but does with Oracle Java 11
Summary: WildFly 20 won't start with our Java 11 but does with Oracle Java 11
Status: RESOLVED FIXED
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: Cauldron
Hardware: All Linux
Priority: Normal normal
Target Milestone: ---
Assignee: Java Stack Maintainers
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-08-21 19:36 CEST by Frank Griffin
Modified: 2021-07-05 16:04 CEST (History)
1 user (show)

See Also:
Source RPM: java-11-openjdk, wildfly-10.1.0-9.mga7.src.rpm
CVE:
Status comment:


Attachments
stdout from failed WildFly (15.75 KB, text/plain)
2020-08-21 19:38 CEST, Frank Griffin
Details
stdout from failed WildFly (5.65 KB, text/plain)
2020-08-22 21:05 CEST, Frank Griffin
Details
stderr from failed WildFly (4.65 KB, text/plain)
2020-08-22 21:10 CEST, Frank Griffin
Details

Description Frank Griffin 2020-08-21 19:36:13 CEST
WildFly is pretty simple to install and start.  Download the TGZ from

   https://download.jboss.org/wildfly/20.0.1.Final/wildfly-20.0.1.Final.tar.gz

create a user, unpack the tar in the home directory, cd to wildfly-20.0.1.Final/bin, and issue

   ./standalone.sh

Using Oracle Java 12, the server comes up (and also, I suspect, with Oracle, Java 11).

Using our new Java 11, it gets immediate errors about trying to create modules using the modular Java support introduced in Java 9.  I'll attach the stacktrace output from the failed run, but I think our Java 11 must be missing some build options having to do with modular support.

Marking as Major since WildFly/JBoss is widely used.
Comment 1 Frank Griffin 2020-08-21 19:38:00 CEST
Created attachment 11818 [details]
stdout from failed WildFly
Comment 2 Lewis Smith 2020-08-21 21:36:57 CEST
Frank, you are doing something iffy by complaining about a 3rd-party application not working.
We *do* seem to offer Wildfly, and normally that is what you should be using. However, running under Mageia 8,
 $ urpmq -i wildfly
 wildfly-10.1.0-9.mga7.src.rpm
so something is wrong. But wildfly is in Cauldron, so we still offer it. Can you say why you are not using it?

CC'ing neoclust who maintains both wildfly & Java 11.

Source RPM: java11 => java-11-openjdk, wildfly-10.1.0-9.mga7.src.rpm
CC: (none) => lewyssmith, mageia
Severity: major => normal

Comment 3 Frank Griffin 2020-08-21 21:45:22 CEST
WildFly 10 is known not to work with Java 11 (for reasons that have nothing to do with module support).  Starting with Java 9, certain coding practices started to get warnings that support for them would be dropped in future releases,  They actually were dropped in Java 10, and thus in Java 11.  WildFly 10 used these practices and failed with Java 11.  I've heard that these were corrected in WildFly 16, but I won't swear to it.

In any case, the point is moot.  A Java implementation should work with any Java application.
Comment 4 Lewis Smith 2020-08-21 22:10:34 CEST
From what you say, I imagine that our current Wildfly 10 should work with java-1.8.0-openjdk which is also in Mageia 8.

> A Java implementation should work with any Java application.
Seems obvious, but Java is a nightmare as your previous comment shows.
Perhaps we can - and should - update our wildfly.

BTAIM assigning this (CC removed) to NicolasL for both packages.

Assignee: bugsquad => mageia
CC: lewyssmith, mageia => (none)

David Walser 2020-08-22 00:05:26 CEST

See Also: (none) => https://bugs.mageia.org/show_bug.cgi?id=27172

David Walser 2020-08-22 00:06:07 CEST

CC: (none) => mageia
Assignee: mageia => java

Comment 5 Frank Griffin 2020-08-22 00:13:09 CEST
@David No, you misunderstand.  WildFly 20 doesn't require Java 12, that's just the >= 11 Oracle version I had on hand to test with.  The problems between Java 11 and WildFly 10 date from Java 10.

If you're more comfortable, I can test with Oracle Java 11, but I'm sure the result will be the same.
Comment 6 Nicolas Lécureuil 2020-08-22 00:13:40 CEST
i think we will remove it from mageia 8
Comment 7 Frank Griffin 2020-08-22 01:20:20 CEST
Just to be clear, I have no objection to removing WildFly from MGA 8.  It is a standalone tar.gz that requires almost no setup, and it is simple for almost anyone who has enough technical cred to want it to set up.  Also, it releases almost constantly.  The preferred way to use it is running under its own ID which can't get to anything else, because it serves mostly network clients who shouldn't be *able* to get to anything else (kind of like Apache), Giving it MGA packaging to park it in official directories is of minimal use, since most of the subdirectories for a running instance have to be writable and unique to that instance.

However, I *do* have an objection to releasing an MGA 8 JDK 11 that can't support Java applications that use the Java Module support, which has been standard since Java 9.  In other words, feel free to drop the MGA WildFly package, but doing that doesn't mean that this bug doesn't have to be fixed.
Comment 8 Nicolas Lécureuil 2020-08-22 11:18:12 CEST
to help. 

Can you extract from the logs the part to check ? ( the error part )
Comment 9 Frank Griffin 2020-08-22 20:02:17 CEST
@Lewis yes, WildFly 10 works with Java 8 (this is what my company has been distributing for several years).  A customer of ours found the problem between 10 and Java 11 when they tried to upgrade.

@Nicolas the stacktraces are in the attachment above.  That's about all there is.
Comment 10 Nicolas Lécureuil 2020-08-22 20:50:05 CEST
can you help to understand better those logs ? 
Where do i see what is missing ? ( to test locally ).

Because just tested locally but it fails for an other reason, port 8080 already used.
Comment 11 Frank Griffin 2020-08-22 21:05:56 CEST
Created attachment 11819 [details]
stdout from failed WildFly

Attachment 11818 is obsolete: 0 => 1

Comment 12 Frank Griffin 2020-08-22 21:10:57 CEST
Created attachment 11820 [details]
stderr from failed WildFly
Comment 13 Frank Griffin 2020-08-22 21:19:13 CEST
Sorry, the first attachment was the stdout from the Java 12 run that worked.  Both attachments are now correct.

From what I can see, the problem is in trying to verify a JARFile.  The lowest level of the stacktrace indicates a security problem but the upper levels indicate a malformed XML file (but doesn't name it).

If you like, I can test with an independently packaged OpenJDK.
Comment 14 Frank Griffin 2020-08-22 21:38:56 CEST
Testing with the official openJDK from https://jdk.java.net/java-se-ri/11 works.
Comment 15 Nicolas Lécureuil 2020-08-22 23:03:05 CEST
maybe a missing deps, i don't have this at all on my mga 8
Comment 16 Nicolas Lécureuil 2020-08-22 23:04:04 CEST
[neoclust@localhost bin]$ ./standalone.sh           
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /home/neoclust/wildfly/wildfly-20.0.1.Final

  JAVA: java

  JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED

=========================================================================

11:15:32,353 INFO  [org.jboss.modules] (main) JBoss Modules version 1.10.1.Final
11:15:32,902 INFO  [org.jboss.msc] (main) JBoss MSC version 1.4.11.Final
11:15:32,913 INFO  [org.jboss.threads] (main) JBoss Threads version 2.3.3.Final
11:15:33,073 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: WildFly Full 20.0.1.Final (WildFly Core 12.0.3.Final) starting
11:15:34,091 INFO  [org.wildfly.security] (ServerService Thread Pool -- 27) ELY00001: WildFly Elytron version 1.12.1.Final
11:15:34,762 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
11:15:34,859 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 37) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
11:15:34,919 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
11:15:34,938 INFO  [org.xnio] (MSC service thread 1-3) XNIO version 3.8.1.Final
11:15:34,964 INFO  [org.xnio.nio] (MSC service thread 1-3) XNIO NIO Implementation Version 3.8.1.Final
11:15:35,048 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 52) WFLYCLINF0001: Activating Infinispan subsystem.
11:15:35,037 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 44) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.4)
11:15:35,099 INFO  [org.jboss.remoting] (MSC service thread 1-5) JBoss Remoting version 5.0.18.Final
11:15:35,110 INFO  [org.wildfly.extension.microprofile.config.smallrye._private] (ServerService Thread Pool -- 61) WFLYCONF0001: Activating WildFly MicroProfile Config Subsystem
11:15:35,107 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 59) WFLYJSF0007: Activated the following JSF Implementations: [main]
11:15:35,121 INFO  [org.jboss.as.jaxrs] (ServerService Thread Pool -- 54) WFLYRS0016: RESTEasy version 3.12.1.Final
11:15:35,135 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 66) WFLYNAM0001: Activating Naming Subsystem
11:15:35,142 INFO  [org.wildfly.extension.microprofile.jwt.smallrye._private] (ServerService Thread Pool -- 63) WFLYJWT0001: Activating WildFly MicroProfile JWT Subsystem
11:15:35,175 INFO  [org.wildfly.extension.microprofile.health.smallrye] (ServerService Thread Pool -- 62) WFLYHEALTH0001: Activating Eclipse MicroProfile Health Subsystem
11:15:35,180 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 72) WFLYSEC0002: Activating Security Subsystem
11:15:35,197 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 76) WFLYWS0002: Activating WebServices Extension
11:15:35,207 INFO  [org.wildfly.extension.microprofile.opentracing] (ServerService Thread Pool -- 65) WFLYTRACEXT0001: Activating MicroProfile OpenTracing Subsystem
11:15:35,150 INFO  [org.jboss.as.connector] (MSC service thread 1-1) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.4.22.Final)
11:15:35,154 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) WFLYJCA0018: Started Driver service with driver-name = h2
11:15:35,212 INFO  [org.jboss.as.security] (MSC service thread 1-7) WFLYSEC0001: Current PicketBox version=5.0.3.Final-redhat-00005
11:15:35,177 INFO  [org.jboss.as.naming] (MSC service thread 1-3) WFLYNAM0003: Starting Naming Service
11:15:35,187 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 74) WFLYTX0013: The node-identifier attribute on the /subsystem=transactions is set to the default value. This is a danger for environments running multiple servers. Please make sure the attribute value is unique.                                                                                                       
11:15:35,245 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 53) WFLYIO001: Worker 'default' has auto-configured to 8 IO threads with 64 max task threads based on your 4 available processors
11:15:35,245 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-1) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]
11:15:35,281 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0003: Undertow 2.1.3.Final starting
11:15:35,309 INFO  [org.jboss.as.ejb3] (MSC service thread 1-6) WFLYEJB0482: Strict pool mdb-strict-max-pool is using a max instance size of 16 (per class), which is derived from the number of CPUs on this host.
11:15:35,316 INFO  [org.wildfly.extension.microprofile.metrics.smallrye] (ServerService Thread Pool -- 64) WFLYMETRICS0001: Activating Eclipse MicroProfile Metrics Subsystem
11:15:35,309 INFO  [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0481: Strict pool slsb-strict-max-pool is using a max instance size of 64 (per class), which is derived from thread worker pool sizing.
11:15:35,462 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 75) WFLYUT0014: Creating file handler for path '/home/neoclust/wildfly/wildfly-20.0.1.Final/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]']
11:15:35,473 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0012: Started server default-server.
11:15:35,512 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0018: Host default-host starting
11:15:35,620 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service org.wildfly.undertow.listener.default: org.jboss.msc.service.StartException in service org.wildfly.undertow.listener.default: Adresse déjà utilisée /127.0.0.1:8080                                                                                                                 








        at java.base/java.lang.Thread.run(Thread.java:834)

11:15:35,752 INFO  [org.jboss.as.ejb3] (MSC service thread 1-7) WFLYEJB0493: EJB subsystem suspension complete
11:15:35,849 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
11:15:35,988 INFO  [org.jboss.as.patching] (MSC service thread 1-4) WFLYPAT0050: WildFly Full cumulative patch ID is: base, one-off patches include: none
11:15:36,013 WARN  [org.jboss.as.domain.management.security] (MSC service thread 1-4) WFLYDM0111: Keystore /home/neoclust/wildfly/wildfly-20.0.1.Final/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost
11:15:36,039 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-8) WFLYDS0013: Started FileSystemDeploymentService for directory /home/neoclust/wildfly/wildfly-20.0.1.Final/standalone/deployments
11:15:36,123 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-1) MSC000001: Failed to start service org.wildfly.undertow.listener.https: org.jboss.msc.service.StartException in service org.wildfly.undertow.listener.https: Adresse déjà utilisée /127.0.0.1:8443
        at org.wildfly.extension.undertow@20.0.1.Final//org.wildfly.extension.undertow.ListenerService.start(ListenerService.java:209)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
        at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
        at java.base/java.lang.Thread.run(Thread.java:834)

11:15:36,209 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
    ("subsystem" => "undertow"),
    ("server" => "default-server"),
    ("http-listener" => "default")
]) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.undertow.listener.default" => "Adresse déjà utilisée /127.0.0.1:8080"}}
11:15:36,211 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
    ("subsystem" => "undertow"),
    ("server" => "default-server"),
    ("https-listener" => "https")
]) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.undertow.listener.https" => "Adresse déjà utilisée /127.0.0.1:8443"}}
11:15:36,219 INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
WFLYCTL0186:   Services which failed to start:      service org.wildfly.undertow.listener.https: Adresse déjà utilisée /127.0.0.1:8443
      service org.wildfly.undertow.listener.default: Adresse déjà utilisée /127.0.0.1:8080
WFLYCTL0448: 1 additional services are down due to their dependencies being missing or failed
11:15:36,280 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
11:15:36,283 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 20.0.1.Final (WildFly Core 12.0.3.Final) started (with errors) in 4401ms - Started 305 of 577 services (3 services failed or missing dependencies, 370 services are lazy, passive or on-demand)
11:15:36,286 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
11:15:36,286 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
Comment 17 Frank Griffin 2020-08-23 14:17:04 CEST
It could be that the in-use ports are preventing the failing components from starting.

I'll try removing and then reinstalling the MGA RPM.
Comment 18 Frank Griffin 2020-08-23 17:40:10 CEST
Removing java-11-openjdk would remove a ton of other stuff, so I settled for urpmi --replacepkgs.  No change.  I'll try installing on another partition.
Comment 19 Nicolas Lécureuil 2020-08-23 17:47:05 CEST
i made sure i do not have openjdk 8 and i think this is OK, i have:


17:44:42,580 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
17:44:42,581 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
Comment 20 Frank Griffin 2020-08-23 19:30:31 CEST
I did a fresh install (checking all categories under Custom), rebooted, and did urpmi java-11-openjdk, which installed fine.  I copied my wflyplog home directory to the new system, verified that java -version gave 11, and got the same errors as reported here.
Comment 21 Frank Griffin 2020-08-23 20:46:11 CEST
Same errors for WildFly 14 Final.
Frank Griffin 2020-08-23 20:47:53 CEST

Summary: WildFly 20 won't start with our Java 11 but does with Oracle Java 12 => WildFly 20 won't start with our Java 11 but does with Oracle Java 11

Comment 22 Nicolas Lécureuil 2020-08-23 22:02:11 CEST
maybe more a missing deps then, as it works for me.
Comment 23 Frank Griffin 2020-08-24 01:52:31 CEST
So where do we go from here ?  I've posted the stacktrace in the WildFly forum, but I hesitate to enter a JIRA since it works with OpenJDK 11, and I doubt that many developers are going to see the forum post.
Comment 24 Nicolas Lécureuil 2020-08-24 10:05:10 CEST
again, as it seems to works for me, this seems to be a missing deps on your machine, we have to find which one but this is definitivly not a wildfly bug to report upstream.
Comment 25 Frank Griffin 2020-08-24 16:01:30 CEST
It's got to be a Java dep, as the WildFlys I'm using are self-contained and not RPMs.  I was hoping someone at WildFly might recognize what it is they're trying to do from the stacktrace.
Comment 26 Frank Griffin 2020-09-01 05:54:06 CEST
(In reply to Nicolas Lécureuil from comment #24)
> again, as it seems to works for me, this seems to be a missing deps on your
> machine, we have to find which one but this is definitivly not a wildfly bug
> to report upstream.

This has nothing to do with my machine, since I did a fresh cauldron install as described in comment#20 and got the same errors.  There is nothing on that new partition that MGA didn't put there.

Also, I still think that the only reason it seems to "work" for you is that the ports that WildFly wants to use are in use by something else on your system, and the parts of Wildfly that get the errors I see never start because of the unavailable ports.
Comment 27 Frank Griffin 2020-09-13 22:52:06 CEST
Update:

I had another issue with WildFly, which led me to reinstall WildFly in its "vanilla" state.  I then executed WildFly in a shell where JAVA_HOME and PATH were pointing to the non-MGA JDK11, and as noted above, it comes up without error.  

When I tried to run jboss-cli to do configuration, I got the errors (on the client, i. e. jboss-cli, side) described above.

On a guess, I tried running jboss-cli in a shell with JAVA_HOME and PATH pointing to the non-MGA JDK11, and it worked fine.  In the error scenario, those variables were pointing to the MGA JDK11.

This is a worse problem than I originally reported, because it means that some applications compiled with or running under MGA JDK11 will fail when interacting with third-party apps built with a non-MGA JDK11 or running under a non-MGA JDK11.
Comment 28 Frank Griffin 2020-09-17 22:13:13 CEST
Based on the stderr attachment above, I'm suspecting that some component of our JDK11 build is incorrectly signed.  This is indicated by the top-level PKCS layers in the stacktrace.
Comment 29 Frank Griffin 2021-07-05 16:04:51 CEST
No longer happening in current cauldron.

Status: NEW => RESOLVED
Resolution: (none) => FIXED


Note You need to log in before you can comment on or make changes to this bug.