Add OIDC single sign-on to any website with HAProxy and Keycloak

Run the unprotected service
Reminder how OIDC works
Creating our client
Manual token fetch with CURL
Redirect to get tokens if there's no session cookie
Implement the token fetch
Store the tokens in a map and set the session cookie
How to troubleshoot an HAProxy configuration
Validate the received tokens
Use profile information in the application
The complete implementation
How could the implementation be improved?
Conclusion

Given a simple web application that is unaware of authentication, it's possible to wrap a single sign-on implementation around it to require authentication using only Keycloak and HAProxy.

Why Keycloak?

Most full-featured identity providers are cloud-based from the large vendors: Okta, OneLogin, AWS, others. The only self-hosted alternative I'm aware of is Ory Hydra which I admittedly haven't tried.

While applications and databases can be migrated to different platforms fairly easily, it's very difficult to migrate your user store to some other product. The vendors may not provide tooling since don't want you to leave, and they probably won't support moving user passwords. This makes the migration high-risk and disruptive for users. For these reasons I advocate self-hosting your IdP, even if the rest of your stack is cloud-native.

Keycloak is also surprisingly good. It does everything including authentication (single sign-on, OIDC) and authorization (OAuth) with a state-of-the-art feature set that I can host myself with no cloud dependencies.

Why HAProxy?

I've been levelling up my HAProxy skills and am always impressed at how good is. I explore a number of new-to-me features in this article.

I suspected that I could take a standalone website and put HAProxy in front of it to require OIDC authentiction.

There are other ways to wrap an application with OIDC like Dex. Using HAProxy was attractive since I'm already running it in a general CDN/WAF way so using it also for OIDC doesn't add a new dependecy to my stack.

It's all running locally. Keycloak is started with the local dev settings:

erik@carbon ~/tmp/keycloak/keycloak-20.0.1 $ ./bin/kc.sh start-dev
2022-12-22 12:56:22,312 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: <request>, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: false
2022-12-22 12:56:23,257 WARN  [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
2022-12-22 12:56:23,883 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2022-12-22 12:56:23,897 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
...

... with a realm "myrealm" and user "myuser" created as per the Get Started guide for bare metal.

The app I'm protecting is the magic8ball example of an HAProxy service. It's not even an application but an HAProxy service which is a small application that runs in the HAProxy process as a Lua script. This code is here.

Run the unprotected service

magic8ball.lua:

local function magic8ball(applet)
    local responses = {"Reply hazy", "Yes - definitely", "Don't count on it", "Outlook good", "Very doubtful"}
    local myrandom = math.random(1, #responses)
    local response = string.[[
        <html>
            <body>
              <div>responses[myrandom]</div>
            </body>
        </html>
    ]]

    applet:set_status(200)
    applet:add_header("content-length", string.8)
    applet:add_header("content-type", "text/html")
    applet:start_response()
    applet:send(response)
end

core.register_service("magic8ball", "http", magic8ball)

This is largely pasted in from the guide linked above.

haproxy.cfg

global
    # service: the target application
    lua-load magic8ball.lua

defaults
    mode http
    timeout connect         10s
    timeout client          1m
    timeout server          1m

frontend fe_main

    bind :8891

    # regular resource
    default_backend resource

backend resource

    # pass legit traffic to application
    http-request use-service lua.magic8ball

This is a boilerplate configuration. Notice the load-lua and use-service directives which wire in our magic 8 ball service.

Now start HAProxy:

erik@carbon ~/tmp/magic8ball $ haproxy -d -f haproxy.cfg
Note: setting global.maxconn to 262126.
Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use epoll.

Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace
Using epoll() as the polling mechanism.
...

The -d flag enables debugging and running in the foreground.

Now the service loads:

We have a trivial service to protect.

Neat that HAProxy provides the capability to run a service. It has other useful hooks like actions, which we'll see in a minute. This makes all kinds of edge computing possible.

Reminder how OIDC works

Tokens (JWTs) are the core of OAuth/OIDC. With some browser redirects and some server-side requests, HAProxy gets a hold of the tokens specific to 'myuser':

The ID token is for our magic8ball application (the OAuth client) to use. It tells us the user is authenticated and has some profile information like the user's full name, etc. Importantly, it doesn't tell us what the user is authorized to do in the applicaiton.
The access token is for authorization. We ignore it here because we're not providing authorization and because our client isn't the consumer ("audience") of this token.
The refresh token lets us get replacement ID and access tokens, allowing the user to remain logged in indefinitely but still securely.

Our main task is to redirect the user to the Keycloak signin page, which will redirect us back to our applicaiton with an authorization code. HAProxy will make a server-side request to get the tokens and store them for the user (keyed to a session ID). Then subsequent requests to the application will check that there's a valid, current ID token for the session so the user is authenticated.

Creating our client

Create a client "magic8ball" in the "myrealm" Keycloak realm:

This client corresponds to our magic8ball application. Most settings can be left at defaults. Some important ones:

The client ID can be any string so "magic8ball" is clear enough.

Add the /code URI to Valid Redirect URIs:

This is an allowlist for where the user can be redircted to after sign-in with the authorization code.

The standard flow is all we require for this implementation.

The client secret has been generated for us. Note its location since we'll need it to fetch the tokens.

Manual token fetch with cURL

This isn't strictly necessary but It's good to check with cURL that we can fetch and inspect the tokens. Then if we get some error later while integrating it into our application we'll have eliminated some variables.

Prepare the basic auth credential. It's base64(clientid:clientsecret). When preparing it like I do here, be careful to press Ctrl+D twice, don't press Enter (so no newline is encoded).

Capture the resulting value (starts with 'bWF' in this example).

Prepare this URL. This is where your application will redirect the browser to in Keycloak for sign-in.

http://localhost:8080/realms/myrealm/protocol/openid-connect/auth?scope=openid&response_type=code&client_id=magic8ball&redirect_uri=http://localhost:8891/code

/realms/myrealm ... our realm
scope=openid ... generally always required to get an ID token
response_type=code ... says we want an authentication code
client_id=magic8ball ... matches the client ID we set
redirect_uri ... the value we allowed in the client configuration

Open the URL in a browser, log in 'myuser', and get redirected to our application. If you didn't leave HAProxy running it's okay. We just want the code value in the redirected URL.

Capture the value of the code query parameter for the next step.

Prepare the token fetch request:

erik@carbon ~ $ curl -X POST -H 'Authorization: Basic bWFnaWM4YmFsbDprU01tYndHOGoxRlVCeG5UZ0FKd1JkdEUwcjBjcnR5Sw==' 'http://localhost:8080/realms/myrealm/protocol/openid-connect/token' --data 'grant_type=authorization_code&client_id=magic8ball&redirect_uri=http://localhost:8891/code&code=379e73c8-5dd2-43cb-9216-2172db4c4b9a.04215bfa-e11c-4d53-bfe6-f3d8ecf94914.c14108a4-e20e-4204-b15f-22798a0ba8d8'
{"access_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJoaHpkV29CWGpDSlVEcjJrOV9INkxGV2p1MEVIVVphUXZya3FFcGdSNWxFIn0.eyJleHAiOjE2NzE5MTY2NDQsImlhdCI6MTY3MTkxNjM0NCwiYXV0aF90aW1lIjoxNjcxOTE2MzA5LCJqdGkiOiI2Y2QzYjczMS1kMjJhLTRhYzEtYTA3MC02ZDAyNjQ4ZTk5NTMiLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjgwODAvcmVhbG1zL215cmVhbG0iLCJhdWQiOlsiYXBpLmNhdHMubmV0IiwiYWNjb3VudCJdLCJzdWIiOiJlZGIxYzM5NS00ZDFhLTQzMjMtOWViMS05YzI1N2FmM2NhMGUiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJtYWdpYzhiYWxsIiwic2Vzc2lvbl9zdGF0ZSI6IjA0MjE1YmZhLWUxMWMtNGQ1My1iZmU2LWYzZDhlY2Y5NDkxNCIsImFjciI6IjEiLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiZGVmYXVsdC1yb2xlcy1teXJlYWxtIiwib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiIsInJlYWxtZm9vcm9sZSJdfSwicmVzb3VyY2VfYWNjZXNzIjp7ImFwaS5jYXRzLm5ldCI6eyJyb2xlcyI6WyJhZG1pbiJdfSwiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJvcGVuaWQgcHJvZmlsZSBlbWFpbCIsInNpZCI6IjA0MjE1YmZhLWUxMWMtNGQ1My1iZmU2LWYzZDhlY2Y5NDkxNCIsImVtYWlsX3ZlcmlmaWVkIjpmYWxzZSwibmFtZSI6Ik15IFVzZXIiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJteXVzZXIiLCJnaXZlbl9uYW1lIjoiTXkiLCJmYW1pbHlfbmFtZSI6IlVzZXIifQ.UD20q7cRzdvvgsX4svkegwOMVwFU6tUZCABSscO2vS1wvy50V7PqjwWmjFn2g1mbMFpas5Vfhlqctkt6frmGzoKuXVs2vOM2oQkRcQH-P210H-J2KOz_QYJZe18XVe3CpujAvXzOf2rwKvcCbpPJppcRkD1BFZymXO1xUN0KdCUppO-X-BAeFPSNePGGFx2z3doWdcA26hkzJgsdT6TIINomsDppj2aXv2kLkvRC4BEjS9rd93VkJMflt6pY3tzKFol6GxD4VZBlF6YQP9EB2oFPZUHjd3mnfEAp36knMQ59HOtcqT27TW-RY-mO14UZmvgxcUXC1llh8hWK--YutA","expires_in":300,"refresh_expires_in":1800,"refresh_token":"eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJkODhhY2ZmZC1lMjhmLTQwYmMtODlhZi1lNGY5YWRhZDJmNTkifQ.eyJleHAiOjE2NzE5MTgxNDQsImlhdCI6MTY3MTkxNjM0NCwianRpIjoiNmUyMTM3YmMtMWVkOS00NDQ0LTg1MjEtMmMzNjRmYWY0YmIzIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwL3JlYWxtcy9teXJlYWxtIiwiYXVkIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwL3JlYWxtcy9teXJlYWxtIiwic3ViIjoiZWRiMWMzOTUtNGQxYS00MzIzLTllYjEtOWMyNTdhZjNjYTBlIiwidHlwIjoiUmVmcmVzaCIsImF6cCI6Im1hZ2ljOGJhbGwiLCJzZXNzaW9uX3N0YXRlIjoiMDQyMTViZmEtZTExYy00ZDUzLWJmZTYtZjNkOGVjZjk0OTE0Iiwic2NvcGUiOiJvcGVuaWQgcHJvZmlsZSBlbWFpbCIsInNpZCI6IjA0MjE1YmZhLWUxMWMtNGQ1My1iZmU2LWYzZDhlY2Y5NDkxNCJ9.TvrHVKesMpjR01BPRiJL1fQ6Fgu_VmkoorQjiqpilYM","token_type":"Bearer","id_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJoaHpkV29CWGpDSlVEcjJrOV9INkxGV2p1MEVIVVphUXZya3FFcGdSNWxFIn0.eyJleHAiOjE2NzE5MTY2NDQsImlhdCI6MTY3MTkxNjM0NCwiYXV0aF90aW1lIjoxNjcxOTE2MzA5LCJqdGkiOiIwMTQ0OTA5OC05MGI1LTRlOTEtYTU1MC0wNGU2ZTJhY2FjNWEiLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjgwODAvcmVhbG1zL215cmVhbG0iLCJhdWQiOiJtYWdpYzhiYWxsIiwic3ViIjoiZWRiMWMzOTUtNGQxYS00MzIzLTllYjEtOWMyNTdhZjNjYTBlIiwidHlwIjoiSUQiLCJhenAiOiJtYWdpYzhiYWxsIiwic2Vzc2lvbl9zdGF0ZSI6IjA0MjE1YmZhLWUxMWMtNGQ1My1iZmU2LWYzZDhlY2Y5NDkxNCIsImF0X2hhc2giOiJqX3Rzc0l2RUpIZ19RVEhSdkQwLUJRIiwiYWNyIjoiMSIsInNpZCI6IjA0MjE1YmZhLWUxMWMtNGQ1My1iZmU2LWYzZDhlY2Y5NDkxNCIsImVtYWlsX3ZlcmlmaWVkIjpmYWxzZSwibmFtZSI6Ik15IFVzZXIiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJteXVzZXIiLCJnaXZlbl9uYW1lIjoiTXkiLCJmYW1pbHlfbmFtZSI6IlVzZXIifQ.FqGK3FfiVAL4SctD5mdT56Ciq42rjnfaq_MAEL6ssL_JtQuKCHma5ObnAKct5TGzCZ6uHGCPgEUcUTYX1gTLqcDQkJPtVnq3losJl2oqossXQ0Zh_eASEDFy7mPXi7C6clqk9kb8xDVc_k3xJ5rJOAsNd7siEL7GJT0sTNhZG3yznLTZjfFwQTaaJMI4hjPLOwof_ti1VbsxlZHJCzAqQ0XpDunUVs-V3FCHhqEdTiOD1j_QbTEtMCevjJxh1g7The8ehJOJgtQjKcGrLcH8MMJEN_U4iIG26pmV7Kgvd5gNvEAb51YNGSRjgcRIl9ZxF3zpqs_rVxgaHBtfUZbuWA","not-before-policy":0,"session_state":"04215bfa-e11c-4d53-bfe6-f3d8ecf94914","scope":"openid profile email"}erik@carbon ~ $

We have tokens! You can inspect them online with the debugger at jwt.io or some console tool like 'cargo install jwt-cli'. Here's the ID token:

erik@carbon ~ $ jwt decode eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJoaHpkV29CWGpDSlVEcjJrOV9INkxGV2p1MEVIVVphUXZya3FFcGdSNWxFIn0.eyJleHAiOjE2NzE3NTk4MTcsImlhdCI6MTY3MTc1OTUxNywiYXV0aF90aW1lIjoxNjcxNzU2ODA1LCJqdGkiOiIwMTQyYTQ0NC0xZjU2LTQ3NDAtYTg4ZS0zZGY0ODc3YmJmY2IiLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjgwODAvcmVhbG1zL215cmVhbG0iLCJhdWQiOiJtYWdpYzhiYWxsIiwic3ViIjoiZWRiMWMzOTUtNGQxYS00MzIzLTllYjEtOWMyNTdhZjNjYTBlIiwidHlwIjoiSUQiLCJhenAiOiJtYWdpYzhiYWxsIiwic2Vzc2lvbl9zdGF0ZSI6ImM3NDQ3ZGVhLTE4MDItNDIyMC1iYTk3LTdmNjMzZjhkMjU1OSIsImF0X2hhc2giOiI4b3RyblJiR2dLbGlPYXFxSnZBeEhBIiwiYWNyIjoiMCIsInNpZCI6ImM3NDQ3ZGVhLTE4MDItNDIyMC1iYTk3LTdmNjMzZjhkMjU1OSIsImVtYWlsX3ZlcmlmaWVkIjpmYWxzZSwibmFtZSI6Ik15IFVzZXIiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJteXVzZXIiLCJnaXZlbl9uYW1lIjoiTXkiLCJmYW1pbHlfbmFtZSI6IlVzZXIifQ.XOIyWMxn77BWSIhIkrts21uZZ-9BdEIUvv8wYoKlfEvEnsp50HYhwyBhsEj6KxEOuhm7rmRu5M5aGArWYV91ARNinUTvbc-ahyVcsQ1FLhB4yU0bWa_3i9-eP9CAx2mnBdbU8u4EdMQFrwKFEF4e7HvRNLtWcKHy1PP3IAZG3YySaG7IqmfuneXr9ITt5yolMVxZPQDfkzRsZh5qdF0ATGjgy__65LWjRhcvSYB15tQVPdrIBj_hiJhBNHXA-pF4W77RQqbRFKXZaJ7g5SD2lGroN6Ir-Wkydmw8SoeRNRJETfHip63GHovNPBP51HshYR7ZWWrJl94WXI9e0kAD9A

Token header
------------
{
  "typ": "JWT",
  "alg": "RS256",
  "kid": "hhzdWoBXjCJUDr2k9_H6LFWju0EHUZaQvrkqEpgR5lE"
}

Token claims
------------
{
  "acr": "0",
  "at_hash": "8otrnRbGgKliOaqqJvAxHA",
  "aud": "magic8ball",
  "auth_time": 1671756805,
  "azp": "magic8ball",
  "email_verified": false,
  "exp": 1671759817,
  "family_name": "User",
  "given_name": "My",
  "iat": 1671759517,
  "iss": "http://localhost:8080/realms/myrealm",
  "jti": "0142a444-1f56-4740-a88e-3df4877bbfcb",
  "name": "My User",
  "preferred_username": "myuser",
  "session_state": "c7447dea-1802-4220-ba97-7f633f8d2559",
  "sid": "c7447dea-1802-4220-ba97-7f633f8d2559",
  "sub": "edb1c395-4d1a-4323-9eb1-9c257af3ca0e",
  "typ": "ID"
}

Redirect to get tokens if there's no session cookie

We will eventually set a cookie "magicsess" that HAProxy can use to look up the tokens from a map. Currently there's no such cookie so the user should be redirected to get the tokens (and later set the session). Here's the haproxy.cfg and a stub magictokens.lua:

global
    # action: exchange an authcode for tokens
    lua-load magictokens.lua

    # service: the target application
    lua-load magic8ball.lua

defaults
    mode http
    timeout connect         10s
    timeout client          1m
    timeout server          1m

frontend fe_main

    bind :8891

    # /code will fetch tokens from an authcode
    use_backend authcode if { path == /code }

    # regular resource
    default_backend resource

backend resource

    # get tokens if no cookie
    acl has_cookie req.cook(magicsess) -m found

    # get tokens
    http-request redirect location 'http://localhost:8080/realms/myrealm/protocol/openid-connect/auth?scope=openid&response_type=code&client_id=magic8ball&redirect_uri=http://localhost:8891/code' unless has_cookie

    # pass legit traffic to application
    http-request use-service lua.magic8ball

backend authcode

    # use this lua action to fetch access/id/refresh tokens with the auth code
    http-request set-var(req.authcode) url_param(code)
    http-request lua.magictokens

    # Send the user somewhere after the tokens are fetched
    http-request redirect location https://cnn.com

    
local function magictokens(txn)

    -------------------------------
    -- Get the auth code variable
    -------------------------------
    authcode = txn.get_var(txn,"req.authcode")
    core.Debug(string.format("auth code is %s",authcode))
end

core.register_action("magictokens",{'http-req'}, magictokens, 0)

Our "resource" backend will redirect to the Keycloak /auth page for signin when the cookie is missing.

After the signin the user is redirected back to the /code URI which goes to the new "authcode" backend.

The "http-request lua.magictokens" executes the Lua "action" in the magictokens.lua file. This action is a stub that just prints the auth code to the debug output. Note the authcode has been parsed and set as a request variable in the 'http-request set-var(req.authcode) url_param(code)" which makes it available to the Lua action.

After the token fetch we'd like to redirect the user back to the /magic URI to get their 8-ball result. For now we redirect to a dummy location (cnn.com) since, without setting a cookie yet, a redirect loop would occur.

Implement the token fetch

magictokens.lua becomes:

local function magictokens(txn)

    -------------------------------
    -- Get the auth code variable
    -------------------------------
    authcode = txn.get_var(txn,"req.authcode")
    --core.Debug(string."auth code is authcode")

    -------------------------------
    -- Prepare token request
    -------------------------------
    -- http://localhost:8080/realms/myrealm/protocol/openid-connect/token

    client_id_and_secret = "bWFnaWM4YmFsbDprU01tYndHOGoxRlVCeG5UZ0FKd1JkdEUwcjBjcnR5Sw=="
    realm = "myrealm"
    token_endpoint = string.format("/realms/%s/protocol/openid-connect/token",realm)
    client_id="magic8ball"
    grant_type="authorization_code"
    redirect_uri="http://localhost:8891/code"
    data=string.format("grant_type=%s&client_id=%s&redirect_uri=%s&code=%s",grant_type,client_id,redirect_uri,authcode)

    auth_header = string.format("Authorization: Basic %s",client_id_and_secret)
    content_length_header = string.format("Content-Length: %d", string.len(data))
    content_type_header = "Content-Type: application/x-www-form-urlencoded"

    token_request = string.format("POST %s HTTP/1.1\r\n%s\r\n%s\r\n%s\r\n\r\n%s",token_endpoint,auth_header,content_type_header,content_length_header,data)

    --core.Debug(string."request is token_request")
    
    -------------------------------
    -- Make request over TCP
    -------------------------------
    contentlen = 0

    idp = core.tcp()
    idp:settimeout(5)
    -- connect to issuer
    if idp:connect('127.0.0.1','8080') then
      if idp:send(token_request) then
      
        -- Skip response headers
        while true do
          local line, err = idp:receive('*l')

          if err then
            core.Alert(string.format("error reading header: %s",err))
            break
          else
            if line then
              --core.Debug(string."data: line")
              if line == '' then break end
              -- core.Debug(string."substr: string.sub(line,1,3)")
              if string.sub(string.lower(line),1,15) == 'content-length:' then
                --core.Debug(string."found content-length: string.sub(line,16)")
                contentlen = tonumber(string.sub(line,16))
              end
            else
              --core.Debug("no more data")
              break
            end
          end
        end

        -- Get response body, if any
        --core.Debug("read body")
        local content, err = idp:receive(contentlen)

        if content then
          -- save the entire token response out to a variable for
          -- further processing in haproxy.cfg
          --core.Debug(string."tokens are content")
          txn.set_var(txn,'req.tokenresponse',content)
        else
          core.Alert(string.format("error receiving tokens: %s",err))
        end
      else
          core.Alert('Could not send to IdP (send)')
      end
      idp:close()
    else
        core.Alert('Could not connect to IdP (connect)')
    end

end

core.register_action("magictokens",{'http-req'}, magictokens, 0)

There's a lot here but it's basically making an HTTP request over a TCP connection, equivalent to the token fetch we did earlier with cURL.

I did HTTP-over-TCP since the Lua blog doc linked above does that, and it's just as awkward as it looks. I noticed later that there's an HTTPClient class in the quite-good documentation. If I did it over I'd use this obviously.

The "txn.set_var(txn,'req.tokenresponse',content)" is saving the JSON blob as a request variable that can be used in the haproxy.cfg after the action completes. I parse the tokens out of the JSON in haproxy.cfg although it should be possible to do this in Lua as well with the Converters class.

Store the tokens in a map and set the session cookie

Maps! These are a powerful and dynamic data structure in the HAProxy process. A good introduction is here.

We want three maps for ID tokens, access tokens and refresh tokens. The key for each map will be our session ID (a random UUID).

Maps require a corresponding file on disk, even if the file is empty and never written to. This seems awkward at first but actually allows persisting the data when HAProxy restarts or reloads, since you can trivially write a cronjob that dumps the runtime data structure onto the disk. Create the empty map files:

erik@carbon ~ $ touch accesstoken.map
erik@carbon ~ $ touch idtoken.map
erik@carbon ~ $ touch refreshtoken.map

local function magicsetcookie(applet)
    applet:set_status(302)

    -- set cookie
    cookie = applet.get_var(applet,'req.sessioncookie')
    --core.Debug(cookie)
    applet:add_header("Set-Cookie", string.format("magicsess=%s; HttpOnly",cookie))

    applet:add_header("Location", "/magic")

    applet:start_response()
end

core.register_service("magicsetcookie", "http", magicsetcookie)

Replace our redirect to cnn.com with another custom Lua service that redirects to /magic and sets the cookie.

(I wanted to do this with an "http-request redirect" directive in haproxy.cfg but it can't set the cookie in the response. This is the workaround.)

global
    # action: exchange an authcode for tokens
    lua-load magictokens.lua

    # service: redirect back to the application while setting a cookie in a way that redirect can't do
    lua-load magicsetcookie.lua

    # service: the target application
    lua-load magic8ball.lua

defaults
    mode http
    timeout connect         10s
    timeout client          1m
    timeout server          1m

frontend fe_main

    bind :8891

    # /code will fetch tokens from an authcode
    use_backend authcode if { path == /code }

    # regular resource
    default_backend resource

backend resource

    # get tokens if no cookie
    acl has_cookie req.cook(magicsess) -m found

    # get tokens
    http-request redirect location 'http://localhost:8080/realms/myrealm/protocol/openid-connect/auth?scope=openid&response_type=code&client_id=magic8ball&redirect_uri=http://localhost:8891/code' unless has_cookie

    # pass legit traffic to application
    http-request use-service lua.magic8ball

backend authcode

    # use this lua action to fetch access/id/refresh tokens with the auth code
    http-request set-var(req.authcode) url_param(code)
    http-request lua.magictokens

    # Generate a random (type 4) UUID as a session key.  This is the key in the maps    
    http-request set-var(req.sessioncookie) uuid

    # This noise is required for our maps to be visible
    # https://github.com/haproxy/haproxy/issues/1156 ... not fixed
    http-request set-header Foo %[path,map(/home/erik/tmp/magic8ball/accesstoken.map)]
    http-request set-header Foo %[path,map(/home/erik/tmp/magic8ball/refreshtoken.map)]
    http-request set-header Foo %[path,map(/home/erik/tmp/magic8ball/idtoken.map)]
    http-request del-header Foo

    # store the fetched tokens in our maps with session ID as key
    http-request set-map(/home/erik/tmp/magic8ball/accesstoken.map) %[var(req.sessioncookie)] %[var(req.accesstoken)]
    http-request set-map(/home/erik/tmp/magic8ball/refreshtoken.map) %[var(req.sessioncookie)] %[var(req.refreshtoken)]
    http-request set-map(/home/erik/tmp/magic8ball/idtoken.map) %[var(req.sessioncookie)] %[var(req.idtoken)]

    # Send the user back to /magic, setting the new session cookie
    http-request use-service lua.magicsetcookie

The comments above describe what the directives do. The "noise" section is an unfortunate workaround for an unresolved bug.

Now I can navigate to my application at "http://localhost:8891/magic", log in if necessary, and have a session created once my tokens are fetched to HAProxy. I can see my "magicsess" cookie created:

We're not done but we have the start of authentication.

How to troubleshoot an HAProxy configuration

Obviously this is a lot. What tools are available to inspect and troubleshoot all this configuration?

Running HAProxy in debug mode

As shown above, running HAProxy with -d gives us debug output and runs in the foreground.

Log customization

HAProxy logging is one line per request. This is what you expect in an HTTP log. When you need println-style debugging it's still possible but takes some extra thought:

global
    ...
    log stdout local2

frontend fe_main

    log global
    # log-format "%[var(sess.now)] %[var(sess.expiration)] ... remaining %[var(sess.now),neg,add(sess.expiration)]"
    # log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
    log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[var(sess.now),neg,add(sess.expiration)] %[var(sess.username)]"
    ...

By setting "log stdout" I can see the HTTP log interleaved with the foreground debug output. Note that HAProxy by design won't touch a filesystem once it has started serving traffic so there's no direct way to say "log to /var/log/haproxy.out". In production you're meant to have syslog running which can write to log files.

The commented log format line above is the default HTTP log format. The uncommented line after it shows that I've added two fields at the end which are session variables that I've set elsewhere in the haproxy.cfg.

"%[var(sess.now),neg,add(sess.expiration)]" - how many seconds are left before the ID token expires
"%[var(sess.username)]" - the username from the ID token

So it's possible to put any information you want in the HTTP log entry.

erik@carbon ~ $ haproxy -d -f haproxy.cfg
Note: setting global.maxconn to 262121.
Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use epoll.

Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace
Using epoll() as the polling mechanism.
00000000:fe_main.accept(0005)=001f from [127.0.0.1:33352] ALPN=
00000000:fe_main.clireq[001f:ffffffff]: GET /magic HTTP/1.1
00000000:fe_main.clihdr[001f:ffffffff]: host: localhost:8891
00000000:fe_main.clihdr[001f:ffffffff]: user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0
00000000:fe_main.clihdr[001f:ffffffff]: accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
00000000:fe_main.clihdr[001f:ffffffff]: accept-language: en-US,en;q=0.5
00000000:fe_main.clihdr[001f:ffffffff]: accept-encoding: gzip, deflate, br
00000000:fe_main.clihdr[001f:ffffffff]: dnt: 1
00000000:fe_main.clihdr[001f:ffffffff]: cookie: magicsess=061d34de-e8ab-4a15-832d-c7eeb30aa8ae
00000000:fe_main.clihdr[001f:ffffffff]: upgrade-insecure-requests: 1
00000000:fe_main.clihdr[001f:ffffffff]: sec-fetch-dest: document
00000000:fe_main.clihdr[001f:ffffffff]: sec-fetch-mode: navigate
00000000:fe_main.clihdr[001f:ffffffff]: sec-fetch-site: cross-site
00000000:resource.clicls[001f:ffff]
00000000:resource.closed[001f:ffff]
<150>Dec 25 09:26:33 haproxy[924472]: 127.0.0.1:33352 [25/Dec/2022:09:26:33.400] fe_main resource/ 0/-1/-1/-1/0 302 239 - - LR-- 1/1/0/0/0 0/0 "GET /magic HTTP/1.1" - -
00000001:fe_main.accept(0005)=001f from [127.0.0.1:33352] ALPN=
00000001:fe_main.clireq[001f:ffffffff]: GET /code?session_state=135ff1b2-0471-497d-a5e5-40ad3a3546b9&code=0f3f1d5f-a2a1-44c4-90a2-0a8fc6dbab60.135ff1b2-0471-497d-a5e5-40ad3a3546b9.c14108a4-e20e-4204-b15f-22798a0ba8d8 HTTP/1.1
00000001:fe_main.clihdr[001f:ffffffff]: host: localhost:8891
00000001:fe_main.clihdr[001f:ffffffff]: user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0
00000001:fe_main.clihdr[001f:ffffffff]: accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
00000001:fe_main.clihdr[001f:ffffffff]: accept-language: en-US,en;q=0.5
00000001:fe_main.clihdr[001f:ffffffff]: accept-encoding: gzip, deflate, br
00000001:fe_main.clihdr[001f:ffffffff]: dnt: 1
00000001:fe_main.clihdr[001f:ffffffff]: cookie: magicsess=061d34de-e8ab-4a15-832d-c7eeb30aa8ae
00000001:fe_main.clihdr[001f:ffffffff]: upgrade-insecure-requests: 1
00000001:fe_main.clihdr[001f:ffffffff]: sec-fetch-dest: document
00000001:fe_main.clihdr[001f:ffffffff]: sec-fetch-mode: navigate
00000001:fe_main.clihdr[001f:ffffffff]: sec-fetch-site: cross-site
00000001:authcode.srvcls[001f:ffff]
00000001:authcode.srvrep[001f:ffffffff]: HTTP/1.1 302 Moved Temporarily
00000001:authcode.srvhdr[001f:ffffffff]: set-cookie: magicsess=9c18a9fb-b855-4a57-aefe-2b6be7dff962; HttpOnly
00000001:authcode.srvhdr[001f:ffffffff]: location: /magic
00000001:authcode.srvhdr[001f:ffffffff]: transfer-encoding: chunked
00000001:authcode.clicls[001f:ffff]
00000001:authcode.closed[001f:ffff]
<150>Dec 25 09:26:33 haproxy[924472]: 127.0.0.1:33352 [25/Dec/2022:09:26:33.434] fe_main authcode/ 19/0/0/0/19 302 153 - - LR-- 1/1/0/0/0 0/0 "GET /code?session_state=135ff1b2-0471-497d-a5e5-40ad3a3546b9&code=0f3f1d5f-a2a1-44c4-90a2-0a8fc6dbab60.135ff1b2-0471-497d-a5e5-40ad3a3546b9.c14108a4-e20e-4204-b15f-22798a0ba8d8 HTTP/1.1" - -
00000002:LUA-SOCKET.clicls[ffff:001e]
00000002:LUA-SOCKET.srvcls[ffff:ffff]
00000002:LUA-SOCKET.closed[ffff:ffff]
00000003:fe_main.accept(0005)=001f from [127.0.0.1:33352] ALPN=
00000003:fe_main.clireq[001f:ffffffff]: GET /magic HTTP/1.1
00000003:fe_main.clihdr[001f:ffffffff]: host: localhost:8891
00000003:fe_main.clihdr[001f:ffffffff]: user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0
00000003:fe_main.clihdr[001f:ffffffff]: accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
00000003:fe_main.clihdr[001f:ffffffff]: accept-language: en-US,en;q=0.5
00000003:fe_main.clihdr[001f:ffffffff]: accept-encoding: gzip, deflate, br
00000003:fe_main.clihdr[001f:ffffffff]: dnt: 1
00000003:fe_main.clihdr[001f:ffffffff]: cookie: magicsess=9c18a9fb-b855-4a57-aefe-2b6be7dff962
00000003:fe_main.clihdr[001f:ffffffff]: upgrade-insecure-requests: 1
00000003:fe_main.clihdr[001f:ffffffff]: sec-fetch-dest: document
00000003:fe_main.clihdr[001f:ffffffff]: sec-fetch-mode: navigate
00000003:fe_main.clihdr[001f:ffffffff]: sec-fetch-site: cross-site
00000003:resource.srvcls[001f:ffff]
00000003:resource.srvrep[001f:ffffffff]: HTTP/1.1 200 OK
00000003:resource.srvhdr[001f:ffffffff]: content-type: text/html
00000003:resource.srvhdr[001f:ffffffff]: content-length: 99
00000003:resource.clicls[001f:ffff]
00000003:resource.closed[001f:ffff]
<150>Dec 25 09:26:33 haproxy[924472]: 127.0.0.1:33352 [25/Dec/2022:09:26:33.457] fe_main resource/ 0/0/0/0/0 200 170 - - LR-- 1/1/0/0/0 0/0 "GET /magic HTTP/1.1" 300 myuser

Note "300 myuser" at the end. This request is for authenticated user "myuser" and the ID token expires in 300 seconds.

Map inspection with the stats socket

We can see the contents of our maps by enabling the stats socket and sending commands to it with the socat program. This is documented above in the maps article.

global
    ...
    # good for inspecting our maps
    stats socket /home/erik/tmp/magic8ball/haproxy.stat mode 600 level admin
    stats timeout 2m

erik@carbon ~ $ socat ~/tmp/magic8ball/haproxy.stat readline
prompt

> show map
# id (file) description
2 (/home/erik/tmp/magic8ball/idtoken.map) pattern loaded from file '/home/erik/tmp/magic8ball/idtoken.map' used by map at file 'haproxy.cfg' line 45, by map at file 'haproxy.cfg' line 50, by map at file 'haproxy.cfg' line 60, by map at file 'haproxy.cfg' line 61, by map at file 'haproxy.cfg' line 78. curr_ver=0 next_ver=0 entry_cnt=1
5 (/home/erik/tmp/magic8ball/accesstoken.map) pattern loaded from file '/home/erik/tmp/magic8ball/accesstoken.map' used by map at file 'haproxy.cfg' line 76. curr_ver=0 next_ver=0 entry_cnt=1
6 (/home/erik/tmp/magic8ball/refreshtoken.map) pattern loaded from file '/home/erik/tmp/magic8ball/refreshtoken.map' used by map at file 'haproxy.cfg' line 77. curr_ver=0 next_ver=0 entry_cnt=1

> show map #2
0x5602fadc7270 9c18a9fb-b855-4a57-aefe-2b6be7dff962 eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJoaHpkV29CWGpDSlVEcjJrOV9INkxGV2p1MEVIVVphUXZya3FFcGdSNWxFIn0.eyJleHAiOjE2NzE5ODU4OTMsImlhdCI6MTY3MTk4NTU5MywiYXV0aF90aW1lIjoxNjcxOTgzMTc2LCJqdGkiOiI1Zjg5MmY4Yy0zNDZhLTRjN2UtYjkyOC1lZDE3NGMzMWM4NTUiLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjgwODAvcmVhbG1zL215cmVhbG0iLCJhdWQiOiJtYWdpYzhiYWxsIiwic3ViIjoiZWRiMWMzOTUtNGQxYS00MzIzLTllYjEtOWMyNTdhZjNjYTBlIiwidHlwIjoiSUQiLCJhenAiOiJtYWdpYzhiYWxsIiwic2Vzc2lvbl9zdGF0ZSI6IjEzNWZmMWIyLTA0NzEtNDk3ZC1hNWU1LTQwYWQzYTM1NDZiOSIsImF0X2hhc2giOiJRWXBIaEJaMVNjcER3a1JyWEVIT2tBIiwiYWNyIjoiMCIsInNpZCI6IjEzNWZmMWIyLTA0NzEtNDk3ZC1hNWU1LTQwYWQzYTM1NDZiOSIsImVtYWlsX3ZlcmlmaWVkIjpmYWxzZSwibmFtZSI6Ik15IFVzZXIiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJteXVzZXIiLCJnaXZlbl9uYW1lIjoiTXkiLCJmYW1pbHlfbmFtZSI6IlVzZXIifQ.EcMYJNBiyRSxneOHapDhbwFm1pmW_7I6c96Gg9gjRnbsni3-OJTmsO3fgxTpvxn_S1GQhzy8pixyvXnoNLCHy2YSKJEqq9QQUmNZPoLWc6sIZiHceo5c7qAjGMcuWKJIjmE-ttfyTlV_ikda3TINo_wMgi8GDVph47q-_Vy_SmQe-E9KtfXnw2lkZSfhiCXOnU6JsFqDesnhUWPYnq2IbXNkXAfUwRMPdUdbfafN3nvQupDTI1MJSPeUWOM2WVBr6NwRgD2gTWJrSaR7B8MeikB2X-ULM2tDquo4dGcxAz7Tor4uP6O32l3awQQqu9sOg0D7ONPPcL9eGZ4UstvaDg

>

Debug statements in Lua

The "core.Debug()" and "core.Alert()" statements in our Lua scripts give us println-style debugging when we're running HAProxy with the -d flag (but not in a production configuration).

Validate the received tokens

Our authentication seems to work but isn't secure yet. A user could set any random UUID as their session cookie and gain access to the application. We must now validate the tokens.

When we fetch the tokens we'll verify the signature, issuer and audience before storing them in our map.

On every request we'll verify that the ID token exists and isn't expired.

Verifying the token signature is done with asymmetric cryptography. We configure the public key beforehand then use it to verify the signature part of the JWT.

For this HAProxy requires having the PEM-encoded keys (JWKS). Keycloak provides a per-realm JSON blob with the keys at http://localhost:8080/realms/myrealm/protocol/openid-connect/certs. There are two not-PEM-encoded keys given so we manually PEM-encode them and configure them at key0.pem and key1.pem and expect the token to be signed by one of them.

Converting the keys from JWKS to PEM was awkward and ultimately I just used this online converter.

Here's the token-fetch-time validation of the signature, issuer and audience. We validate the signature of the access token too though technically we're not the consumer (audience) of the access token so it's not our problem to validate that.

backend authcode
...
    # validate token signature - access token
    http-request set-var(req.accesstoken) var(req.tokenresponse),json_query('$.access_token')
    http-request set-var(req.jwt_alg) var(req.accesstoken),jwt_header_query('$.alg')
    acl jwt_sig_rs256 var(req.jwt_alg) -m str -i "RS256"
    http-request deny if !jwt_sig_rs256
    acl key0_valid var(req.accesstoken),jwt_verify(req.jwt_alg,"/home/erik/tmp/magic8ball/key0.pem") 1
    acl key1_valid var(req.accesstoken),jwt_verify(req.jwt_alg,"/home/erik/tmp/magic8ball/key1.pem") 1
    http-request deny if !key0_valid !key1_valid

    # don't validate token signature - refresh token - note it's HS256 and doesn't validate with PEM files?
    http-request set-var(req.refreshtoken) var(req.tokenresponse),json_query('$.refresh_token')

    # validate token signature - id token
    http-request set-var(req.idtoken) var(req.tokenresponse),json_query('$.id_token')
    http-request set-var(req.jwt_alg) var(req.idtoken),jwt_header_query('$.alg')
    acl jwt_sig_rs256 var(req.jwt_alg) -m str -i "RS256"
    http-request deny if !jwt_sig_rs256
    acl key0_valid var(req.idtoken),jwt_verify(req.jwt_alg,"/home/erik/tmp/magic8ball/key0.pem") 1
    acl key1_valid var(req.idtoken),jwt_verify(req.jwt_alg,"/home/erik/tmp/magic8ball/key1.pem") 1
    http-request deny if !key0_valid !key1_valid

    # validate id token issuer
    acl id_token_issuer_valid var(req.idtoken),jwt_payload_query('$.iss') -m str "http://localhost:8080/realms/myrealm"
    http-request deny if !id_token_issuer_valid

    # validate id token audience - should be our client ID
    acl id_token_audience_valid var(req.idtoken),jwt_payload_query('$.aud') -m str "magic8ball"
    http-request deny if !id_token_audience_valid
...

Here's the validation done every request to our protected resource. We expect the ID token to exist in our map and to not be expired. If those things don't all validate we redirect to fetch new tokens (as before), possibly prompting the user to log in again.

backend resource

    # get tokens if no cookie
    acl has_cookie req.cook(magicsess) -m found

    # get tokens if cookie doesn't map to an id token
    acl cookie_has_id_token req.cook(magicsess),map(/home/erik/tmp/magic8ball/idtoken.map) -m found

    # get tokens if id token is expired
    #   (eventually http-request lua.magicrefresh if invalid exp)
    http-request set-var(sess.now) date
    http-request set-var(sess.expiration) req.cook(magicsess),map(/home/erik/tmp/magic8ball/idtoken.map),jwt_payload_query('$.exp','int')
    acl id_token_not_expired var(sess.now),neg,add(sess.expiration) gt 0

    # get tokens
    http-request redirect location 'http://localhost:8080/realms/myrealm/protocol/openid-connect/auth?scope=openid&response_type=code&client_id=magic8ball&redirect_uri=http://localhost:8891/code' unless has_cookie cookie_has_id_token id_token_not_expired

We have now secured our application.

Use profile information in the application

We're now free to pass information from the ID token to the application if we want, though this isn't required. Here we set variables sess.name and sess.username which our magic8ball service can render.

backend resource
...
    # TODO: retool /code to also use refresh token to get new tokens

    # want to pass headers X-Authenticated-User and X-Name for use in the application
    # but a Lua service doesn't get them.  Using variables instead
    http-request set-var(sess.username) req.cook(magicsess),map(/home/erik/tmp/magic8ball/idtoken.map),jwt_payload_query('$.preferred_username')
    http-request set-var(sess.name) req.cook(magicsess),map(/home/erik/tmp/magic8ball/idtoken.map),jwt_payload_query('$.name')
    # pass legit traffic to application
    http-request use-service lua.magic8ball

local function magic8ball(applet)
    -- If client is POSTing request, receive body
    -- local request = applet:receive()

    local responses = {"Reply hazy", "Yes - definitely", "Don't count on it", "Outlook good", "Very doubtful"}
    local myrandom = math.random(1, #responses)
    local response = string.format([[
        <html>
            <body>
              <div>%s</div>
              <div>For you, %s</div>
            </body>
        </html>
    ]], responses[myrandom], applet.get_var(applet,"sess.name"))

    applet:set_status(200)
    applet:add_header("content-length", string.len(response))
    applet:add_header("content-type", "text/html")
    applet:start_response()
    applet:send(response)
end

core.register_service("magic8ball", "http", magic8ball)

The complete implementation

Here's the complete implementation.

haproxy.cfg:

global

    # action: exchange an authcode for tokens
    lua-load magictokens.lua

    # service: redirect back to the application while setting a cookie in a way that redirect can't do
    lua-load magicsetcookie.lua

    # service: the target application
    lua-load magic8ball.lua

    # good for inspecting our maps
    stats socket /home/erik/tmp/magic8ball/haproxy.stat mode 600 level admin
    stats timeout 2m

    log stdout local2

defaults
    mode http
    timeout connect         10s
    timeout client          1m
    timeout server          1m

frontend fe_main

    log global
    # log-format "%[var(sess.now)] %[var(sess.expiration)] ... remaining %[var(sess.now),neg,add(sess.expiration)]"
    # log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
    log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[var(sess.now),neg,add(sess.expiration)] %[var(sess.username)]"

    bind :8891

    # /code will fetch tokens from an authcode
    use_backend authcode if { path == /code }

    # regular resource
    default_backend resource

backend resource

    # get tokens if no cookie
    acl has_cookie req.cook(magicsess) -m found

    # get tokens if cookie doesn't map to an id token
    acl cookie_has_id_token req.cook(magicsess),map(/home/erik/tmp/magic8ball/idtoken.map) -m found

    # get tokens if id token is expired
    #   (eventually http-request lua.magicrefresh if invalid exp)
    http-request set-var(sess.now) date
    http-request set-var(sess.expiration) req.cook(magicsess),map(/home/erik/tmp/magic8ball/idtoken.map),jwt_payload_query('$.exp','int')
    acl id_token_not_expired var(sess.now),neg,add(sess.expiration) gt 0

    # get tokens
    http-request redirect location 'http://localhost:8080/realms/myrealm/protocol/openid-connect/auth?scope=openid&response_type=code&client_id=magic8ball&redirect_uri=http://localhost:8891/code' unless has_cookie cookie_has_id_token id_token_not_expired

    # TODO: retool /code to also use refresh token to get new tokens

    # want to pass headers X-Authenticated-User and X-Name for use in the application
    # but a Lua service doesn't get them.  Using variables instead
    http-request set-var(sess.username) req.cook(magicsess),map(/home/erik/tmp/magic8ball/idtoken.map),jwt_payload_query('$.preferred_username')
    http-request set-var(sess.name) req.cook(magicsess),map(/home/erik/tmp/magic8ball/idtoken.map),jwt_payload_query('$.name')
    # pass legit traffic to application
    http-request use-service lua.magic8ball

backend authcode

    # use this lua action to fetch access/id/refresh tokens with the auth code
    http-request set-var(req.authcode) url_param(code)
    http-request lua.magictokens

    # Generate a random (type 4) UUID as a session key.  This is the key in the maps    
    http-request set-var(req.sessioncookie) uuid

    # This noise is required for our maps to be visible
    # https://github.com/haproxy/haproxy/issues/1156 ... not fixed
    http-request set-header Foo %[path,map(/home/erik/tmp/magic8ball/accesstoken.map)]
    http-request set-header Foo %[path,map(/home/erik/tmp/magic8ball/refreshtoken.map)]
    http-request set-header Foo %[path,map(/home/erik/tmp/magic8ball/idtoken.map)]
    http-request del-header Foo

    # validate token signature - access token
    http-request set-var(req.accesstoken) var(req.tokenresponse),json_query('$.access_token')
    http-request set-var(req.jwt_alg) var(req.accesstoken),jwt_header_query('$.alg')
    acl jwt_sig_rs256 var(req.jwt_alg) -m str -i "RS256"
    http-request deny if !jwt_sig_rs256
    acl key0_valid var(req.accesstoken),jwt_verify(req.jwt_alg,"/home/erik/tmp/magic8ball/key0.pem") 1
    acl key1_valid var(req.accesstoken),jwt_verify(req.jwt_alg,"/home/erik/tmp/magic8ball/key1.pem") 1
    http-request deny if !key0_valid !key1_valid

    # don't validate token signature - refresh token - note it's HS256 and doesn't validate with PEM files?
    http-request set-var(req.refreshtoken) var(req.tokenresponse),json_query('$.refresh_token')

    # validate token signature - id token
    http-request set-var(req.idtoken) var(req.tokenresponse),json_query('$.id_token')
    http-request set-var(req.jwt_alg) var(req.idtoken),jwt_header_query('$.alg')
    acl jwt_sig_rs256 var(req.jwt_alg) -m str -i "RS256"
    http-request deny if !jwt_sig_rs256
    acl key0_valid var(req.idtoken),jwt_verify(req.jwt_alg,"/home/erik/tmp/magic8ball/key0.pem") 1
    acl key1_valid var(req.idtoken),jwt_verify(req.jwt_alg,"/home/erik/tmp/magic8ball/key1.pem") 1
    http-request deny if !key0_valid !key1_valid

    # validate id token issuer
    acl id_token_issuer_valid var(req.idtoken),jwt_payload_query('$.iss') -m str "http://localhost:8080/realms/myrealm"
    http-request deny if !id_token_issuer_valid

    # validate id token audience - should be our client ID
    acl id_token_audience_valid var(req.idtoken),jwt_payload_query('$.aud') -m str "magic8ball"
    http-request deny if !id_token_audience_valid

    # store the fetched tokens in our maps with session ID as key
    http-request set-map(/home/erik/tmp/magic8ball/accesstoken.map) %[var(req.sessioncookie)] %[var(req.accesstoken)]
    http-request set-map(/home/erik/tmp/magic8ball/refreshtoken.map) %[var(req.sessioncookie)] %[var(req.refreshtoken)]
    http-request set-map(/home/erik/tmp/magic8ball/idtoken.map) %[var(req.sessioncookie)] %[var(req.idtoken)]

    # Send the user back to /magic, setting the new session cookie
    http-request use-service lua.magicsetcookie

magic8ball.lua

local function magic8ball(applet)
    -- If client is POSTing request, receive body
    -- local request = applet:receive()

    local responses = {"Reply hazy", "Yes - definitely", "Don't count on it", "Outlook good", "Very doubtful"}
    local myrandom = math.random(1, #responses)
    local response = string.format([[
        <html>
            <body>
              <div>%s</div>
              <div>For you, %s</div>
            </body>
        </html>
    ]], responses[myrandom], applet.get_var(applet,"sess.name"))

    applet:set_status(200)
    applet:add_header("content-length", string.len(response))
    applet:add_header("content-type", "text/html")
    applet:start_response()
    applet:send(response)
end

core.register_service("magic8ball", "http", magic8ball)

magicsetcookie.lua

local function magicsetcookie(applet)
    applet:set_status(302)

    -- set cookie
    cookie = applet.get_var(applet,'req.sessioncookie')
    --core.Debug(cookie)
    applet:add_header("Set-Cookie", string.format("magicsess=%s; HttpOnly",cookie))

    applet:add_header("Location", "/magic")

    applet:start_response()
end

core.register_service("magicsetcookie", "http", magicsetcookie)

magictokens.lua

local function magictokens(txn)

    -------------------------------
    -- Get the auth code variable
    -------------------------------
    authcode = txn.get_var(txn,"req.authcode")
    --core.Debug(string."auth code is authcode")

    -------------------------------
    -- Prepare token request
    -------------------------------
    -- http://localhost:8080/realms/myrealm/protocol/openid-connect/token

    client_id_and_secret = "bWFnaWM4YmFsbDprU01tYndHOGoxRlVCeG5UZ0FKd1JkdEUwcjBjcnR5Sw=="
    realm = "myrealm"
    token_endpoint = string.format("/realms/%s/protocol/openid-connect/token",realm)
    client_id="magic8ball"
    grant_type="authorization_code"
    redirect_uri="http://localhost:8891/code"
    data=string.format("grant_type=%s&client_id=%s&redirect_uri=%s&code=%s",grant_type,client_id,redirect_uri,authcode)

    auth_header = string.format("Authorization: Basic %s",client_id_and_secret)
    content_length_header = string.format("Content-Length: %d", string.len(data))
    content_type_header = "Content-Type: application/x-www-form-urlencoded"

    token_request = string.format("POST %s HTTP/1.1\r\n%s\r\n%s\r\n%s\r\n\r\n%s",token_endpoint,auth_header,content_type_header,content_length_header,data)

    --core.Debug(string."request is token_request")
    
    -------------------------------
    -- Make request over TCP
    -------------------------------
    contentlen = 0

    idp = core.tcp()
    idp:settimeout(5)
    -- connect to issuer
    if idp:connect('127.0.0.1','8080') then
      if idp:send(token_request) then
      
        -- Skip response headers
        while true do
          local line, err = idp:receive('*l')

          if err then
            core.Alert(string.format("error reading header: %s",err))
            break
          else
            if line then
              --core.Debug(string."data: line")
              if line == '' then break end
              -- core.Debug(string."substr: string.sub(line,1,3)")
              if string.sub(string.lower(line),1,15) == 'content-length:' then
                --core.Debug(string."found content-length: string.sub(line,16)")
                contentlen = tonumber(string.sub(line,16))
              end
            else
              --core.Debug("no more data")
              break
            end
          end
        end

        -- Get response body, if any
        --core.Debug("read body")
        local content, err = idp:receive(contentlen)

        if content then
          -- save the entire token response out to a variable for
          -- further processing in haproxy.cfg
          --core.Debug(string."tokens are content")
          txn.set_var(txn,'req.tokenresponse',content)
        else
          core.Alert(string.format("error receiving tokens: %s",err))
        end
      else
          core.Alert('Could not send to IdP (send)')
      end
      idp:close()
    else
        core.Alert('Could not connect to IdP (connect)')
    end

end

core.register_action("magictokens",{'http-req'}, magictokens, 0)

How could this implementation be improved?

While secure and useful, there are more steps to make this production-ready and maintainable.

TLS

... in production, obviously, for both Keycloak and HAProxy.

Use Lua HTTP Client class

As noted above, there's an easier way to do HTTP from the Lua action.

Use the refresh token

When the ID token expires the user is redirected back to Keycloak and may have to log in again. We can instead use the refresh token on the server side to fetch new tokens. I'd modify the magictokens action to support this case too. This would provide the user a less-disruptive experience.

Add authorization

We have implemented authentication in a way that any realm user can access the application. What if we want to require that users have a certain role or are in a certain group to interact with certain resources. For this Keycloak provides the access token which has a list of permissions that. Enabling authorization is another whole thing but is totally possible. In that case we'd expose those permissions from the access token to the application to base authorization decisions.

Socket task: periodically dump tokens to map files

If HAProxy is reloaded or restarted, the in-memory contents of the maps are lost. As hinted above, a cronjob can periodically dump the map contents to the map files using the socket (socat).

Socket task: periodically remove expired tokens

Over time many session UUID's are created and put into the maps, growing unbounded. A cronjob, again using the socket (socat), can sweep the map and delete entries with expired tokens so the data remains lean.

Periodically download JWKS keys

We manually downloaded the JWKS keys and converted them to PEM format. Of course those keys expire eventually. A cronjob can download them periodically, convert to PEM, and reload the HAProxy process when they've changed.

Conclusion

I've shown how an auth-unaware web application can be made to require authentication using Keycloak and HAProxy alone.

I'd be lying if I said I whipped that out in a couple hours. It was a lot of hours, enough that it was worth documenting in a blog post.

Still, I'd absolutely reach for this solution in a case where some first- or third-party application needs to be made to authenticate with an organization's standard OIDC solution (in this case Keycloak).

I want to also show off how powerful those two technologies are.

I hope you found this interesting and maybe useful.

12/30/2022 by stasibear Permalink

oauth oidc keycloak haproxy