Hi,
Our VMWare environment has been running for a couple of years now and all of a sudden we're getting an error when opening the URL of our non-default tenant "ICT". The error message is: "Login failed. Please contact your system administrator" and a randomly generated code. I did some Googling and most of the results for this generic error message lead to NTP or certificate related causes, both of which I have checked and are configured just fine. The default tenant works without any problem for both local users (administrator@vsphere.local) as domain users (both vCenter and vRA have an identity store towards our Microsoft Domain Controllers). Also these settings have not been changed. I know it sounds silly, but I tried rebooting all of the impacted appliances, but without any change.
I did some digging around in the Catalina.out (on vRA) file and stumbled on the following:
2019-01-28 11:07:07,518 vcac: [component="cafe:shell" priority="INFO" thread="tomcat-http--3" tenant="ICT"] com.vmware.identity.websso.client.endpoint.SsoRequestSender.getRequestUrl:160 - Producing redirect url
2019-01-28 11:07:07,522 vcac: [component="cafe:shell" priority="INFO" thread="tomcat-http--3" tenant="ICT"] com.vmware.identity.websso.client.endpoint.SsoRequestSender.createRenewable:309 - Added Renewable condition
2019-01-28 11:07:07,522 vcac: [component="cafe:shell" priority="INFO" thread="tomcat-http--3" tenant="ICT"] com.vmware.identity.websso.client.endpoint.SsoRequestSender.createDelegable:317 - Added Delegable condition
2019-01-28 11:07:07,523 vcac: [component="cafe:shell" priority="INFO" thread="tomcat-http--3" tenant="ICT"] com.vmware.identity.websso.client.endpoint.SsoRequestSender.getRequestUrl:252 - Destination URL: https://vl-vcs.domain.local:7444/websso/SAML2/SSO/ICT
2019-01-28 11:07:07,527 vcac: [component="cafe:shell" priority="INFO" thread="tomcat-http--3" tenant="ICT"] com.vmware.identity.websso.client.endpoint.SsoRequestSender.createRequestString:393 - Message to be signed: SAMLRequest=zVVdb9sgFP0rFu82tuMuGapTdcmqRWrXrPamaS8TwTcNEgYPcJz8%2B4HzsbRqq6hPk%2FwEl8M55x6uL682tQjWoA1XMkdJFKMAJFMVl485%2Bl7ehCN0Nb40tBYNuW7tSj7AnxaMDdw5aUi%2FkaNWS6Ko4YZIWoMhlpHi%2Bu6WpFFMGq2sYkqgYOrOcUltf9XK2sYQjNciXDMTCcWoiOxK1dTUSkO0ADLMsgx3sDBGYQ%2BX4qK4x7NJiYIbpRn0fHK0pMIACmbTHP2GLP2QUsiSi8FgOKqWcZJmo2yYDmmcxQvm1M3MnBrD1%2FDvoDEtzKSxVNocpXHyMYyTMB2VSUziofuiizT%2BhYL5XsgnLnf2vKV6sSsy5EtZzsP5fVH2AGtegf7qqnPUC6fsJeWYM4uCH4euODx06EFPVp%2FvvtMK2juOxk8cf%2BViv4GVfvQuYw9%2F8L8GSytq6SU%2B5bFjlTbES5pN50pwtj0hl56djWshVDfRQK2zxuoW%2Bh7X1L4N4Fd4FS77UtJ4x4wF6dwr5p7Tt5YKvuSgnwXuvfIR3ksmE%2BUa7I01p3rPbsZzlD3I2h3ZUfVM6446XkzV2LAVOJ6YWqvDHhi7oKY4zvDnjRPsg2LQHmRj%2BBGj67qoG0ReUxrHCf55d1v0WCHvE8%2Bc0a6e2G3jM%2BkZP4CEji4ElG7tBcH%2FEdUpCHg8pYqfN%2BeY0P3cgqqfGq7Awsa%2BK6oTVTdUc%2BOfJmyoe6q7S8gp8kS4Zj%2FA8uSGs7PxZhkjzEO7ZT%2FHOqUrP5eAOWWlptI0StudDS%2FyOVj0iiFHuwqmGje%2F3mWQI7TZTlTrx2ly7MrT%2F8f4Lw%3D%3D&RelayState=aHR0cHM6Ly92bC12Y2FjLmxvY2FsLnRob21hc21vcmUuYmUvdmNhYy9vcmcvSUNULw%3D%3D&SigAlg=http%3A%2F%2Fwww.w3.org%2F2001%2F04%2Fxmldsig-more%23rsa-sha256
2019-01-28 11:07:07,547 vcac: [component="cafe:shell" priority="INFO" thread="tomcat-http--3" tenant="ICT"] com.vmware.identity.websso.client.endpoint.SsoRequestSender.getRequestUrl:263 - Redirect URL: https://vl-vcs.domain.local:7444/websso/SAML2/SSO/ICT?SAMLRequest=zVVdb9sgFP0rFu82tuMuGapTdcmqRWrXrPamaS8TwTcNEgYPcJz8%2B4HzsbRqq6hPk%2FwEl8M55x6uL682tQjWoA1XMkdJFKMAJFMVl485%2Bl7ehCN0Nb40tBYNuW7tSj7AnxaMDdw5aUi%2FkaNWS6Ko4YZIWoMhlpHi%2Bu6WpFFMGq2sYkqgYOrOcUltf9XK2sYQjNciXDMTCcWoiOxK1dTUSkO0ADLMsgx3sDBGYQ%2BX4qK4x7NJiYIbpRn0fHK0pMIACmbTHP2GLP2QUsiSi8FgOKqWcZJmo2yYDmmcxQvm1M3MnBrD1%2FDvoDEtzKSxVNocpXHyMYyTMB2VSUziofuiizT%2BhYL5XsgnLnf2vKV6sSsy5EtZzsP5fVH2AGtegf7qqnPUC6fsJeWYM4uCH4euODx06EFPVp%2FvvtMK2juOxk8cf%2BViv4GVfvQuYw9%2F8L8GSytq6SU%2B5bFjlTbES5pN50pwtj0hl56djWshVDfRQK2zxuoW%2Bh7X1L4N4Fd4FS77UtJ4x4wF6dwr5p7Tt5YKvuSgnwXuvfIR3ksmE%2BUa7I01p3rPbsZzlD3I2h3ZUfVM6446XkzV2LAVOJ6YWqvDHhi7oKY4zvDnjRPsg2LQHmRj%2BBGj67qoG0ReUxrHCf55d1v0WCHvE8%2Bc0a6e2G3jM%2BkZP4CEji4ElG7tBcH%2FEdUpCHg8pYqfN%2BeY0P3cgqqfGq7Awsa%2BK6oTVTdUc%2BOfJmyoe6q7S8gp8kS4Zj%2FA8uSGs7PxZhkjzEO7ZT%2FHOqUrP5eAOWWlptI0StudDS%2FyOVj0iiFHuwqmGje%2F3mWQI7TZTlTrx2ly7MrT%2F8f4Lw%3D%3D&RelayState=aHR0cHM6Ly92bC12Y2FjLmxvY2FsLnRob21hc21vcmUuYmUvdmNhYy9vcmcvSUNULw%3D%3D&SigAlg=http%3A%2F%2Fwww.w3.org%2F2001%2F04%2Fxmldsig-more%23rsa-sha256&Signature=Red1i3EPrEz8NVDNNt44zhfz13ulpFKA7jsj4IMTYwqf99UWnVBI5ZZlSmSpq8kndKNgA%2BM5EkNb8SLgC%2F2sh1L7fQ%2BxLEvELRFgLOoU6iEDKEVUKRZhnKSPzWoF2RiQYFg0%2FnRgGyBJfQE0jYPWet9v4lwdXGEY0OfpItJTwkAGxXzZhHDHdQO6Q3R57o24t4M0Rx5Ql9CRBn9taYtFajTf5HPDnOF7gzo7h9hFUb66lr71olCcXX0B5q0TQBewLInuVrpcAiPAuai2Jit4H2YbF71iWTZT3ZJQXPeHR6OvsAZ8Fn8MrP0RiEQGLwGdww5m0gnOqd68YW%2FDtlycLA%3D%3D
2019-01-28 11:07:07,551 vcac: [component="cafe:shell" priority="INFO" thread="tomcat-http--3" tenant="ICT"] com.vmware.identity.websso.client.MessageStoreImpl.add:221 - New MessageStore entry added:%s , store size: %s
2019-01-28 11:07:07,885 vcac: [component="cafe:shell" priority="INFO" thread="tomcat-http--3" tenant="ICT"] com.vmware.identity.websso.client.endpoint.SsoResponseListener.consumeResponse:77 - You have POST'ed to Websso client library!
2019-01-28 11:07:07,885 vcac: [component="cafe:shell" priority="INFO" thread="tomcat-http--3" tenant="ICT"] com.vmware.identity.websso.client.SsoValidationState.validate:72 - Validating SAMLResponse..
2019-01-28 11:07:07,895 vcac: [component="cafe:shell" priority="INFO" thread="tomcat-http--3" tenant="ICT"] com.vmware.identity.websso.client.ValidationState.validateDestination:64 - Validating request destination: HttpservletRequest destination=https://vl-vcac.domain.local/vcac/org/ICT/saml/websso/ssoSAML message destination=https://vl-vcac.domain.local/vcac/org/ICT/saml/websso/sso
2019-01-28 11:07:07,896 vcac: [component="cafe:shell" priority="INFO" thread="tomcat-http--3" tenant="ICT"] com.vmware.identity.websso.client.SsoValidationState.validateInResponseTo:158 - Validating optional request ID: _e4262ae4153378df012484727a040bc0
2019-01-28 11:07:07,897 vcac: [component="cafe:shell" priority="INFO" thread="tomcat-http--3" tenant="ICT"] com.vmware.identity.websso.client.SsoValidationState.validate:108 - Received failure response from SSO server, status code: urn:oasis:names:tc:SAML:2.0:status:Responder substatus code: urn:oasis:names:tc:SAML:2.0:status:RequestDenied
2019-01-28 11:07:07,897 vcac: [component="cafe:shell" priority="INFO" thread="tomcat-http--3" tenant="ICT"] com.vmware.identity.websso.client.MessageStoreImpl.add:221 - New MessageStore entry added:%s , store size: %s
2019-01-28 11:07:07,898 vcac: [component="cafe:shell" priority="ERROR" thread="tomcat-http--3" tenant="ICT"] com.vmware.vcac.authentication.http.SamlLogonProcessor.authenticationError:72 - SAMLResponse received, but failed to parse and validate it. Source: https://vl-vcs.domain.local:7444/websso/SAML2/Metadata/ICT, destination: https://vl-vcac.domain.local/vcac/org/ICT/saml/websso/sso, Issued: 2019-01-28T10:07:07.645Z, Status: urn:oasis:names:tc:SAML:2.0:status:Responder, Reason: null
2019-01-28 11:07:07,899 vcac: [component="cafe:shell" priority="ERROR" thread="tomcat-http--3" tenant="ICT"] com.vmware.vcac.authentication.http.SamlAssertionAuthenticationFilter.attemptAuthentication:81 - WebSSO finished processing SAML Authentication at '/vcac/org/ICT/saml/websso/sso' abnormally.No token found after SAMLResponse processing. Please see previous lines for more information about the reason and SSO log files.
2019-01-28 11:07:07,900 vcac: [component="cafe:shell" priority="ERROR" thread="tomcat-http--3" tenant="ICT"] com.vmware.vcac.authentication.http.SamlAssertionAuthenticationFilter.unsuccessfulAuthentication:124 - Unsuccessful Authentication for '/vcac/saml/websso/sso'. Reason:'Can not authenticate the user, no credentials were provided.'
2019-01-28 11:07:07,900 vcac: [component="cafe:shell" priority="ERROR" thread="tomcat-http--3" tenant="ICT"] com.vmware.vcac.authentication.http.LoginErrorEntryPoint.commence:82 - Exception with error code Y8WdM2MX:
org.springframework.security.authentication.BadCredentialsException: Can not authenticate the user, no credentials were provided.
at com.vmware.vcac.authentication.http.SamlAssertionAuthenticationFilter.attemptAuthentication(SamlAssertionAuthenticationFilter.java:87)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:211)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at com.vmware.vcac.authentication.http.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:39)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at com.vmware.vcac.authentication.http.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:39)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at com.vmware.vcac.authentication.http.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:39)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at com.vmware.vcac.authentication.http.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:39)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at com.vmware.vcac.authentication.http.CspHostEnforceFilter.doFilter(CspHostEnforceFilter.java:57)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:144)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at com.vmware.vcac.authentication.http.tenancy.TenancyContextFilter.doFilterWithTenancyContext(TenancyContextFilter.java:67)
at com.vmware.vcac.authentication.http.SameTenantContextFilter.doFilterWithTenancyContext(SameTenantContextFilter.java:72)
at com.vmware.vcac.authentication.http.tenancy.TenancyContextFilter.doFilter(TenancyContextFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:486)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:411)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:338)
at org.tuckey.web.filters.urlrewrite.NormalRewrittenUrl.doRewrite(NormalRewrittenUrl.java:195)
at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:159)
at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:141)
at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:90)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:417)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
I tried Googling on some of the messages, but didn't find anything useful. Does anyone have an idea how I can solve this?
Thanks in advance!