oAuth2 client with password grant in Spring Security

I'm consuming a set of oAuth2 protected services. It currently works like this: the client logs in using their username and password. I exchange these for a token. I keep the token in the session and submit it every time I want to call a service. It works, but the problem is that I do this completely manually, without using much of Spring Security. Here's how it looks:

<!-- Configure Authentication mechanism -->
<authentication-manager alias="authenticationManager">
    <authentication-provider ref="oAuth2AuthenticationProvider"/>
</authentication-manager>


<beans:bean id="oAuth2AuthenticationProvider" class="my.custom.Oauth2AuthenticationProvider">
    <beans:constructor-arg name="accessTokenUri" value="http://ift.tt/1G5hjUE"/>
    <beans:constructor-arg name="clientId" value="myClientId"/>
    <beans:constructor-arg name="clientSecret" value="myClientSecret"/>
    <beans:constructor-arg name="scope">
        <beans:list>
            <beans:value>myScope</beans:value>
        </beans:list>
    </beans:constructor-arg>
</beans:bean>

<beans:bean id="resourceOwnerPasswordAccessTokenProvider" class="org.springframework.security.oauth2.client.token.grant.password.ResourceOwnerPasswordAccessTokenProvider"/>

As you can see I made the authentication provider myself. It is accepting the standard UsernamePasswordAuthenticationToken but is producing my own extension of it that keeps the actual OAuth2AccessToken as well, thus keeping in the security context.

public class Oauth2AuthenticationProvider implements AuthenticationProvider {

@Autowired
private ResourceOwnerPasswordAccessTokenProvider provider;

private String accessTokenUri;
private String clientId;
private String clientSecret;
private List<String> scope;

public Oauth2AuthenticationProvider(String accessTokenUri, String clientId, String clientSecret, List<String> scope) {
    this.accessTokenUri = accessTokenUri;
    this.clientId = clientId;
    this.clientSecret = clientSecret;
    this.scope = scope;
}

@Override
public Authentication authenticate(Authentication authentication) throws AuthenticationException {
    String username = authentication.getName();
    String password = authentication.getCredentials().toString();
    OAuth2AccessToken token = obtainToken(username, password);
    return handleLogonSuccess(authentication, token);
}

private OAuth2AccessToken obtainToken(String username, String password) {
    ResourceOwnerPasswordResourceDetails passwordResourceDetails = new ResourceOwnerPasswordResourceDetails();
    passwordResourceDetails.setUsername(username);
    passwordResourceDetails.setPassword(password);
    passwordResourceDetails.setClientId(clientId);
    passwordResourceDetails.setClientSecret(clientSecret);
    passwordResourceDetails.setScope(scope);
    passwordResourceDetails.setAccessTokenUri(accessTokenUri);
    DefaultAccessTokenRequest defaultAccessTokenRequest = new DefaultAccessTokenRequest();
    OAuth2AccessToken token;
    try {
        token = provider.obtainAccessToken(passwordResourceDetails, defaultAccessTokenRequest);
    } catch (OAuth2AccessDeniedException accessDeniedException) {
        throw new BadCredentialsException("Invalid credentials", accessDeniedException);
    }

    return token;
}

public OAuth2AccessToken refreshToken(OAuth2AuthenticationToken authentication) {
    OAuth2AccessToken token = authentication.getoAuth2AccessToken();
    OAuth2RefreshToken refreshToken = token.getRefreshToken();
    BaseOAuth2ProtectedResourceDetails resourceDetails = new BaseOAuth2ProtectedResourceDetails();
    resourceDetails.setClientId(clientId);
    resourceDetails.setClientSecret(clientSecret);
    resourceDetails.setScope(scope);
    resourceDetails.setAccessTokenUri(accessTokenUri);
    OAuth2AccessToken newToken = provider.refreshAccessToken(resourceDetails, refreshToken, new DefaultAccessTokenRequest());
    authentication.setoAuth2AccessToken(newToken);
    return newToken;
}

public boolean supports(Class<?> authentication) {
    return UsernamePasswordAuthenticationToken.class.isAssignableFrom(authentication);
}

private Authentication handleLogonSuccess(Authentication authentication, OAuth2AccessToken token) {

    MyCustomOAuth2AuthenticationToken successAuthenticationToken = new MyCustomOAuth2AuthenticationToken(user, authentication.getCredentials(), user.getAuthorities(), token);

    return successAuthenticationToken;
}

public void setProvider(ResourceOwnerPasswordAccessTokenProvider provider) {
    this.provider = provider;
}

}

As you see, it basically makes sure the token remains in the security scope from where I can simply extract it manually before each call to the back-end services. similarly, I will check the freshness of the token before each call. This works well, but I'm sure I can use Spring's oauth namespace in XML (I'm not using Java config) to achieve the same in a more-config-less-code way. The problem with all the examples is either that is includes the oAuth server implementation which I do not care about and just confuse me or that the client is instantiated once with the username/password specified in the XML. This i don't get because those should be provided dynamically by the user...

Can anyone please help me with this?