Connecting Auth0 to Power BI with Stream Analytics and a Webtask

My previous blog post briefly covered a few use cases for Webtask and showed how to use a Webtask to transfer Auth0 logs to Application Insights.

In this post we'll take it a step further and show how you can connect your Auth0 logs to Power BI. This would allow you to create advanced reports where you can see how a marketing campaign on Twitter would impact signups in your application.

Shipping your data to Blob Storage

Wait what? Blob Storage? Wasn't this about Power BI and stuff. Yes, but the plan is the following:

1. We transfer the Auth0 logs to Blob Storage
2. Stream Analytics processes these files (input)
3. Stream Analytics sends them to Power BI (output)

So your first job is to create a Storage Account if you don't have on already and then also configure your Webtask account if you haven't used it before (this uses Node.js):

npm i -g wt-cli  
wt init  

The code for the Webtask is trivial, it just takes your Auth0 logs and creates 1 Blob for each record. The full source is available on GitHub.

Run the following command to schedule the task:

wt cron schedule \  
    --name auth0-logs-to-blob-storage \
    --secret AUTH0_DOMAIN="{YOUR_AUTH0_DOMAIN}" \
    --secret AUTH0_GLOBAL_CLIENT_ID="{YOUR_AUTH0_GLOBAL_CLIENT_ID}" \
    --secret AUTH0_GLOBAL_CLIENT_SECRET="{YOUR_AUTH0_GLOBAL_CLIENT_SECRET}" \
    --secret STORAGE_ACCOUNT_NAME="{YOUR_STORAGE_ACCOUNT_NAME}" \
    --secret STORAGE_ACCOUNT_KEY="{YOUR_STORAGE_ACCOUNT_KEY}" \
    --secret STORAGE_CONTAINER_NAME="{YOUR_STORAGE_CONTAINER_NAME}" \
    --json \
    "*/5 * * * *" \
    https://raw.githubusercontent.com/sandrinodimattia/auth0-logs-to-blob-storage-webtask/master/task.js

This command will get the code from my repository and run it every 5 minutes. The code itself is smart enough to only create Blobs for the new log entries. If you want to make a change to the code you can just fork the repository and change the url in the command.

Now you'll need to replace the variables with the settings for your Storage Account and Auth0 Account (the Global Client Id and Secret can be found in the API Explorer).

Run wt logs to see the output of every run:

Setting up Stream Analytics

The next step is to create a new Stream Analytics Job:

The input of this job will be the Storage Account to where the Auth0 logs are being shipped:

Note the Path pattern, the Date format and Time format. Make sure you use the same settings when you configure your own job. This is required because the logs will be exported in the following "directory structure":

https://{YOUR_STORAGE_ACCOUNT_NAME}.blob.core.windows.net/{YOUR_STORAGE_CONTAINER_NAME}/YYYY/MM/DD/HH/{LOG_ID}.json  

This covers the input (retrieval of the blobs). Now we'll need to configure the output to Power BI. This isn't supported in the Ibiza Portal yet, so you'll need to execute this following step in the old portal instead.

Give the dataset a name and a table (over time you might create different outputs with other tables, to handle different queries).

As a last step in the Stream Analytics configuration you'll need to write a query that takes data from the input and writes it to the output:

SELECT  
  type,
  count(*)
INTO  
  [logs-to-powerbi]
FROM  
  [auth0-logs] TIMESTAMP BY date 
GROUP BY TumblingWindow(minute, 1), type  

This simple query will for example count how many times the different events (eg: Sign Up, Login, ...) occurred.

Now go ahead and start the Stream Analytics job.

Creating your report in Power BI

After a few minutes you'll see the Dataset showing up in your Power BI workspace. From there you'll be able to add the information to your reports and integrate this with your other metrics and data sources.

Enjoy!