Thursday, August 11, 2022

OAuth Integration - Challenges

What seemed like a simple endeavor -- learn how to implement OAuth to protect web services and portal applications using Java and Spring -- turned out to be much more complicated than expected. The posts in this series are targeting two groups -- developers with little background in the goals and mechanics of Oauth and more experienced developers who just need help avoiding pitfalls resulting from inconsistencies in implementations of OAuth. The content is divided into multiple posts:

OAuth Integration - Overview
OAuth Integration - Challenges<---- YOU ARE HERE
OAuth Integration - Authorization Server
OAuth Integration - Command Line Validation
OAuth Integration - Resource Server
OAuth Integration - Service Client
OAuth Integration - User Portal Application


Normalizing Roles / Claims / Authorizations

Each remote Authorization Server is likely to implement different schemes for assigning roles to users and labeling those roles with strings. An application allowing authentication / authorization to more than one remote Authorization Server must be able to create unique mappings for each remote provider used to map them back to a consistent internal view to enforce internal authorization restrictions. As a simple example, a GUI application might need to allow authentication through two providers mdhlabs and providerx and normalize their external roles to a single set of internal roles as shown below.

mdhlabs Roleproviderx RoleInternal Role
mdhlabsuserproviderxbasiccustomer
mdhlabsfullproviderxadvancedadmin

The mapping mechanism not only has to provide the flexibility for this token mapping but must also accommodate all of these variations:

  1. role information likely contains unique values per authorization provider
  2. role information is often mapped to different claim names by different authorization providers
  3. some providers return role information in the access token, others only provide it in a separate user-info endpoint response
  4. many public authorization providers like Google, Facebook, LinkedIn and GitHub don't implement any roles within their applications (because users are all individuals, not members of a group) so no meaningful role information is returned

For example, Keycloak defaults to returning role information as an array under /realm_access/roles in Json replies in access tokens (while also returning it in user-info responses). It can also be configured to return them as a duplicate top level element (as configured for mdhalbs) under any desired name. Here are two replies side by side reflecting that difference.

mdhlabs Roleproviderx Role
"iat": 1654106542, "jti": "8f4862f2-dd04-49e2-854a-c5d00744b5fa", "iss": "http://192.168.99.10:8011/realms/mdhlabs", "aud": "account", "sub": "dependsgetonly", "typ": "Bearer", "azp": "KuberDependsOAuth", "session_state": "52cffb42-e26e-41f1-bd5b-9099bccb509c", "allowed-origins": [ "http://192.168.99.10:8080" ], "realm_access": { "roles": [ "offline_access", "mdhlabsgetonly", "uma_authorization", "default-roles-mdhlabs" ] }, "resource_access": { "account": { "roles": [ "manage-account", "manage-account-links", "view-profile" ] } }, "scope": "email profile", "sid": "52cffb42-e26e-41f1-bd5b-9099bccb509c", "email_verified": false, "roles": [ "offline_access", "mdhlabsgetonly", "uma_authorization", "default-roles-mdhlabs" ], "name": "Eric Clapton", "preferred_username": "dependsgetonly", "given_name": "Eric", "family_name": "Clapton", "email": "slowhand@rsorecords.com" } "iat": 1657223439, "auth_time": 1657222682, "jti": "59196b0c-4aec-4925-9136-365bf0ac2ca1", "iss": "http://192.168.99.10:8011/realms/providerx", "aud": "account", "sub": "3435f1ea-dfac-40ae-9f4e-e47d0cd3952a", "typ": "Bearer", "azp": "DependsGUI", "session_state": "ed90dffb-ba5d-4caa-a154-56a1a155a2c7", "acr": "0", "allowed-origins": [ "http://192.168.99.10:8022" ], "realm_access": { "roles": [ "default-roles-providerx", "offline_access", "providerxadvanced", "uma_authorization" ] }, "resource_access": { "account": { "roles": [ "manage-account", "manage-account-links", "view-profile" ] } }, "scope": "openid email profile", "sid": "ed90dffb-ba5d-4caa-a154-56a1a155a2c7", "email_verified": true, "name": "Pete Townsend", "preferred_username": "petetownsend", "given_name": "Pete", "family_name": "Townsend", "email": "petetownsend@trackrecords.com" }


Inconsistent Authorization Server Implementations

In as few words as possible, while people think of OAuth as a standard for implementing authorization flows, it is at best a standard of CONCEPTS, not of actual implementation details. The same community that devised OAuth later added a higher level set of abstractions for encoding actual AUTHORIZATION DETAILS called Open ID Connect, attempting to provide uniformity for parameter names of specific authorization data being shared to simplify encoding / parsing of requests / responses. In short, both efforts have failed miserably.

After implementing integrations to various platforms and providers offering Oauth2 and Open ID Connect based integration, the variances between those implementations are summarized below in table form. This table will be valuable as code modules are designed to arbitrate these variances while trying to maintain uniformity at each stage of authorization.

If developing web services or clients only involving a specific authorization Server provider, these variations don't pose much of an implementation concerns. Identifying the server used, capturing its response formats then coding to them provides the integration needed.

If developing a human facing portal application that needs to integrate to MULTIPLE external Authorization Server providers, variations in response payloads must be absorbed and normalized by the application since the providers' responses are typically unconfigurable since they serve tens of millions of users and hundreds of existing integration partners. Those variations are summarized below for the providers used in these examples.

These payloads are not EXACT in all cases. Some long opaque token strings have been shorted to minimize space. Some providers pack the response JSON, others format with linefeeds in the JSON. However, the logical structures shown are exact.


Provider Access Token Content User-Info Content
Facebook {"access_token":"EAAR9qFQDlongopaquetokenZDZD", "token_type":"bearer", "expires_in":5168856} {"id":"5445088938904508", "name":"Oliver Douglas", "email":"oliver.douglas\u0040yahoo.com"}
Github {"access_token":"gho_MEeQeRQ5UDdcAT6vCa2O3Kws", "token_type":"bearer", "scope":""} {"login":"mdhlabs","id":105172001,"node_id":"U_kgDOBkTMIQ", (bunch of URL parms) "type":"User", "site_admin":false, "name":"Oliver Douglas", "company":"mdhLabs, Inc.", "blog":"" ,"location":"Podunk, AR USA","email":null, "hireable":null, "bio":null, "twitter_username":null, "public_repos":0, "public_gists":0, "followers":0, "following":0, "created_at":"2022-05-08T19:02:09Z", "updated_at":"2022-07-28T01:43:53Z"}
Google { "access_token": "ya29.longopaquetokenleqQ0165", "expires_in": 3599, "scope": "https://www.googleapis.com/auth/userinfo.profile openid https://www.googleapis.com/auth/userinfo.email", "token_type": "Bearer", "id_token": "eyJhbGcibase64userinfohere SwmeScszzxHpEZj5AxRL-nDQ" } { "iss": "https://accounts.google.com", "azp": "1040605327873-vlteu.apps.googleusercontent.com", "aud": "1040605327873-vljeu.apps.googleusercontent.com", "sub": "107652139365923884059", "email": "oliverdouglas@gmail.com", "email_verified": true, "at_hash": "j74I0R63gXq9yVKq1s-V_Q", "name": "oliver", "picture": "https://lh3.googleusercontent.com/a/AItbv0l1xvRH_FA=s96-c", "given_name": "Oliver", "locale": "en", "iat": 1659399286, "exp": 1659402886 }
Keycloak { "exp": 1659399542, "iat": 1659399422, "auth_time": 1659399421, "jti": "341514e8-a4e2-428e-810c-0856e83e9c15", "iss": "http://fedora1.mdhlabs.com:8011/realms/mdhlabs", "aud": "account", "sub": "6fd27288-ef5a-44f6-9bfc-23ad324f209d", "typ": "Bearer", "azp": "DependsGUI", "session_state": "b2f1f990-9c91-4c52-aeb9-1fe058e5b46d", "allowed-origins": [ "https://fedora1.mdhlabs.com:8443" ], "realm_access": { "roles": [ "offline_access", "uma_authorization", "default-roles-mdhlabs", "mdhlabsuser" ] }, "resource_access": { "account": { "roles": [ "manage-account", "manage-account-links", "view-profile" ] } }, "scope": "openid email profile", "sid": "b2f1f990-9c91-4c52-aeb9-1fe058e5b46d", "email_verified": true, "roles": [ "offline_access", "uma_authorization", "default-roles-mdhlabs", "mdhlabsuser" ], "name": "Eric Clapton", "groups": [ "offline_access", "uma_authorization", "default-roles-mdhlabs", "mdhlabsuser" ], "preferred_username": "ericclapton", "given_name": "Eric", "family_name": "Clapton", "email": "eric.clapton@rsorecords.com" } {"sub":"6fd27288-ef5a-44f6-9bfc-23ad324f209d", "email_verified":true, "name":"Eric Clapton", "groups":["offline_access", "uma_authorization", "default-roles-mdhlabs","mdhlabsuser"], "preferred_username":"ericclapton", "given_name":"Eric", "family_name":"Clapton", "email":"eric.clapton@rsorecords.com"} LinkedIn {"access_token":"AQWjrlongopaquetokenIBHbpX", "expires_in":5184000} {"localizedLastName":"Douglas", "firstName":{"localized": {"en_US":"Oliver"}, "preferredLocale":{"country":"US","language":"en"}}, "lastName":{"localized": {"en_US":"Douglas"}, "preferredLocale":{"country":"US","language":"en"}}, "id":"OMclyzWGLH", "localizedFirstName":"Oliver"}


Spring Security Library Evoluation

As of 2022, the state of software libraries and online support and examples is highly confusing. This is partly due to the development history of the different OAuth capabilities and poor choices in terminology and library names by their architects. It is useful to understand some of the development evolution to avoid latching onto code examples using concepts that are either deprecated or in conflict with one another which can result in needless work or improper functionality.

Oauth was publicly launched in 2012 and in the case of Java, library support for Oauth functionality was fragmented across different open source development teams and commercial vendors:

  • baseline Spring Security libraries added logic to implement client flows for obtaining authorization tokens and access tokens and handling refreshes of tokens
  • new standalone Spring Security Oauth libraries (not part of Spring Security) were developed to create standardized interface classes for security filter functions as well as packages and annotations to create Authorization Server functionality and Resource Server functionality.
  • external vendors created commercial implementations of Authorization Server functionalty which supported integrated LDAP / RDBMS / cache platforms or integration to existing, widely used public Identity Providers and equivalents commonly used within large companies (Active Directory LDAP, PING Directory LDAP). Some of these solutions (like Keycloak) developed their own proprietary libraries to simplify implementation of Resource Server filtering in client applicationsThese solutions offered their own libraries that could be used to implement the Resource Server functionality. While providing a faster implementation, these proprietary libraries were not interoperable across different platforms and would require major refactoring if a different Authorization Server provider was later adopted.
  • OpenID standards evolved from version 1.0 to 2.0 then were refactored AGAIN and renamed OpenID Connect. The difference between "OpenID" and "OpenID Connect" may be glossed over in many onine help references but all Open ID v1 and OpenID v2 libraries have been deprecated.

The CRUCIAL POINT here is that around 2018, the Spring community recognized the inefficiency of these overlapping efforts and made three key changes:

  1. development in Spring Security Oauth libraries was FROZEN and all existing functionality flagged as DEPRECATED
  2. development of Authorization Server functionality was FROZEN in recognition of this functionality being better developed as a standalone product by commercial firms and independent parties
  3. development of Resource Server functionality was relocated to the core Spring Security project as of version 5.2

These changes have CRUCIAL impacts to developers wanting to implement OAuth:

  1. Coding your own Authorization Server implementation with the older Spring Security OAuth libraries is STRONGLY DISCOURAGED. Find a commercial offering such as Keycloak, Okla, Auth0, etc. and use that instead.
  2. Avoid using vendor specific libraries at the Resource Server layer to "simplify" integration to a third party Authorization Server. Time saved up front will be swamped by refactoring required later as that vendor platform is updated over time or if it is jetissoned in factor of another vendor down the road.
  3. When searching for example solutions, etc during development, ensure any example under review is not based upon an internal implementation of an Authorization Server (#1) or use of vendor-specific librariers (#2).
  4. When searching for technical help, extra caution is required to avoid accidentally injecting use of deprecated classes associated with "OpenID" or classes in older Spring Security OAuth2 libraries rather than the newer consolidated Spring Security libraries.


General Spring Library Evolution

Beyond libraries specific to OAuth, libraries related to general application security, management of Http client connections and SpringBoot optimizations have evolved significantly since roughly 2018. The evoutions aren't a problem but they pose SIGNIFICANT challenges when trying to search for code examples or fixes for specific problems. Searches will INEVITABLY fall on examples using older deprecated libraries that will drag in other deprecated dependencies unless extreme caution is taken. The examples in this series of posts for OAuth will attempt to use the most recent standardized libraries and methodologies. Cases where conflicts and confusion may crop up will be highlighted in specific sections.


WebSecurityConfigurerAdapter (deprecated) Versus SecurityFIlterChain (directional)

Configuration behavior at servlet startup has evolved significantly as of Spring Security version 5.4. In general, Spring Security implements protections by executing a series of filters against each incoming request. Essentially, each filter focuses on performing a particular taks that contributes to

  • deciding to explicitly block a request
  • deciding to explicitly permit a request
  • examining request content (URIs, headers, cookies, etc.) as part of authentication / authorization

The Spring Security framework provides at least 23 different filters focsed on different phases of security processing in the lifetime of a request. Conceptually, they execute in the following order when present but not all of the classes are added unless explicitly configured or implicitly configured by including other Spring libraries which are detected at startup.

ChannelProcessingFilter
ConcurrentSessionFilter
SecurityContextPersistenceFilter
LogoutFilter
X509AuthenticationFilter
AbstractPreAuthenticatedProcessingFilter
CasAuthenticationFilter
UsernamePasswordAuthenticationFilter
ConcurrentSessionFilter
OpenIDAuthenticationFilter
DefaultLoginPageGeneratingFilter
ConcurrentSessionFilter
DigestAuthenticationFilter
BasicAuthenticationFilter
RequestCacheAwareFilter
SecurityContextHolderAwareRequestFilter
JaasApiIntegrationFilter
RememberMeAuthenticationFilter
AnonymousAuthenticationFilter
SessionManagementFilter
ExceptionTranslationFilter
FilterSecurityInterceptor
SwitchUserFilter

Developers can also create unique classes extending the GenericFilterBean class and insert them anywhere within this sequence to meet application-specific needs. At application startup, all defined members of the security filter chain will be listed in an INFO message like that below captured from a Resource Server web service implementation.

2022-07-12 09:27:19,622 INFO org.springframework.security.web.DefaultSecurityFilterChain - Will secure any request with [org.springframework.security.web.session.DisableEncodeUrlFilter@45394b31, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@1ec7d8b3, org.springframework.security.web.context.SecurityContextPersistenceFilter@3eb631b8, org.springframework.security.web.header.HeaderWriterFilter@c68a5f8, org.springframework.security.web.authentication.logout.LogoutFilter@41477a6d, com.mdhlabs.oauthsecurity.mdhlabsJwtFilter@555cf22, org.springframework.security.web.authentication.AuthenticationFilter@3b0ca5e1, org.springframework.security.oauth2.server.resource.web.BearerTokenAuthenticationFilter@736ac09a, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@6bff19ff, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@41e1455d, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@5bb3131b, org.springframework.security.web.session.SessionManagementFilter@3aefae67, org.springframework.security.web.access.ExceptionTranslationFilter@4642b71d, org.springframework.security.web.access.intercept.AuthorizationFilter@591fd34d]

The recommended process for defining the behavior of these filters or adding custom filters was materially re-engineered beginning with Spring Security 5.4 released on September 9, 202. While similar at some level, the two approaches are NOT compatible with one another so any accidental use of code patterns from both in a single project will produce odd problems at compile time or start-up.

Library versions prior to 5.4 included three standard class with these names / functionality:

AuthorizationServerConfigurerAdapter Used to configure code that will act as an Authorization Server using Spring Security OAuth libraries. DEPRECATED. Should not be used.
ResourceServerConfigurerAdapter Used to configure code implementing web services that will use Resource Server patterns to apply endpoint protections. This will typically be used for web services that will be called by other machine processes.
WebSecurityConfigurerAdapter Used to configure code implementing servlet applications or reactive applications that require control of user-facing authentication flows in addition to protection of underlying web services used for presentation layers.

Prior to 5.4, applications would configure the filter chain by

  1. creating a custom SecurityConfiguration class extending one of the above three classes and using @Configuration and @EnableWebSecurity to cause Spring to scan the components at startup to link them together
  2. creating an @Override annotated method with void configure(HttpSecurity http)
  3. using sub-methods of the HttpSecurity object to configure specific default filters or insert new custom filters.

Beginning with Spring Security 5.4, use of the ____ConfigurerAdapter classes is deprecated and will likely be removed in Spring Security 6.x. From version 5.4 and beyond, filtering configuration is accomplished by the following steps

  1. creating a pure custom SecurityConfiguration class (no extends of _____ConfigurerAdapter) and using @Configuration and @EnableWebFluxSecurity (not @EnableWebSecurity) to cause Spring to scan the components at startup to link them together
  2. creating a @Bean method in that class with signature public SecurityChainFilter filterChain(HttpSecurity http)
  3. using sub-methods of the HttpSecurity object to configure specific default filters or insert new custom filters
  4. ending that method by returning http.build() as the response value


RestTemplate (deprecated) Versus WebClient (directional)

In the larger Spring library, the existing RestTemplate class used to simplify integration to remote REST web service endpoints was deprecated in 5.7 in favor of WebClient which was enhanced to support newer stream oriented paradigms in addition to traditional synchronous ("blocking") calls that wait until a response is received or timeout limits reached.

The switch from RestTemplate to WebClient impacts development of OAuth2 clients not only for the interaction with the desired service being protected but also because the client will also use WebClient to interact with the Authorization Server to obtain access tokens or refresh them. That means the Spring libraries needed to create ALL NEW classes and method names to reflect the use of WebClient-oriented functions instead of the older RestTemplate functions. This can generate ENORMOUS confusion because code examples on public sites written years ago will still show old class and method names which will generate never-ending compilation conflicts.

For convenience, a cross reference of old and new classes needed for the deprecated and direction approaches are summarized below.

RestTemplate (deprecated) WebClient (directional)
OAuth2AuthorizedClientProviderReactiveOAuth2AuthorizedClientProvider
DefaultOAuth2AuthorizedClientManagerDefaultReactiveOAuth2AuthorizedClientManager
ClientRegisteryRepositoryReactiveClientRegistryRepository
HttpServletRequestServerHttpRequest
HttpServletResponseServerHttpResponse


Spring MVC Versus WebFlux Applications

Because OAuth functionality involves hooks into libraries that make separate web service calls to Authorization Servers to manage access tokens, developers need to understand differences in application design between the older Spring MVC servlet model and the newer Spring WebFlux model. Older MVC libraries are thread pool based and block those threads even when initiating asynchronous flows to remote endpoints. The newer WebFlux model implements an event loop model that dispatches events once available to compute threads. WebFlux suppors a wider variety of interaction patterns between local and remote processes:

  • traditional synchronous blocking wait
  • traditional outbound asynchronous non-blocking wait
  • new outbound streaming producer
  • new inbound streaming listener

While the extra abstraction is more efficient in scenarios with high request volumes with variable response characteristics, the abstractions are more complex and are not easily updated from older MVC concepts without major refactoring.

These differences are important when developing OAuth2 integrations because the revamped OAuth2 functions within Spring Security default to using WebClient as the connection tool to Authorization Servers and remote endpoints instead of the older RestTemplate. Because the non-blocking behavior of WebFlux is so distinct from older thread-based functions in RestTemplate functionality, many aspects of WebClient development are mutually exclusive to tradtional RestTemplate methods. There are several critical impacts to these changes:

  1. For non-Spring Boot applications, the spring-web and spring-webflux libraries are mutually exclusive and cannot be present in the same project. If both are accidentally included, filters associated with spring-web will be loaded first and anything related to spring-webflux will be ingored.
  2. For Spring Boot applications, the spring-boot-starter-web and spring-boot-starter-webflux libraries are mutually exclusive and NORMALLY should NOT be present in the same project. If both are accidentally included, traditional MVC behavior will be configured via spring-boot-starter-web will be configured and anything related to spring-boot-starter-webflux will be ingored.
  3. HOWEVER, because recent OAuth2 libraries use WebClient for interacting with Authorization Server providers and protected web servie endpoints, they will require WebFlux libraries for the WebClient support. That will force the application by default to ignore WebFlux configurations and deploy as a traditional MVC application. If the application has some dependency on old MVC functions but is being written as a Reactive application, Spring must be explicitly forced to deploy the application as Reactive by setting spring.main.web-application-type=reactive in the application.properties file.
  4. If an application needs to be deployed with a navigation scheme of http://myhost.domain.com/context/path/servlet/path where prior configurations would set server.contextpath=/context/path and server.servletpath=/servlet/path, the application will need to add filtering to manually insert the /context/path for each request.
Posted at