Build a local dashboard to overview your different pipelines
Following several project simultaneously, whether it is for work or personal project can be a little bit tricky to follow easily. Here we will walk through the creation of a local and free dashboard to keep track of the different projects and the status of the different pipelines within them. If you feel comfortable with CI providers rest API, you can easily extend the following example and adapt it to your needs.
Introduction
I am often working on different project at the same time, both for work and private ones. Though I often face problems when monitoring them. In order to keep track I would normally need to do several things:
- Log in to the CI/CD provider provider
- Find the project
- Find the build/release pipeline.
Doing that, I often find myself juggling with numerous number of window tabs, and I quickly start mixing up tabs.
To make things even easier, the different projects I work with are unfortunately scattered across different CI providers and different teams/organizations…
The setup here probably does not happen for most, but I imagine I am definitely not the only one, especially if you are in charge of maintaining multiple products/applications.
The goal here will then be to build a dashboard to gather the results of our pipelines in different projects scattered around in different CI providers, thanks to the different rest API they offer.
What are our options?
In order to build this dashboard, there could be different alternatives: 🔍
- We could of course go for a commercial solutions. I did not personally look into those to be honest, but I guess there has to be something out there in this context.
- Use an open source dashboard such as Grafana for example, and push the results to some database: The advantage is that we would also get the history over time and could store different types of data and display it in different ways via graphs, charts etc.
- Build a dashboard ourselves.
The goal is not to reinvent the wheel, but all I want here is to have a dashboard that shows the state of the different pipelines I need to monitor. More importantly we want to have fun and experiment with a couple of CI providers rest APIs!
For our tech stack, we will use Angular with Typescript to make our journey easier, but other web application framework like ExpressJS would work really.
Set up a new project
Alright, let’s walk through the implementation of this dashboard.
Make sure first of all that you have node js installed along with angular-cli. Personally I am using node v14.15.x and angular v11.x, but as long as you use something close to that you will be fine,
First thing first, let’s create our angular project. We will call it angular-dashboard.
To gain time we will skip the routing module and apply scss templates.
ng new angular-dashboard --strict true --style scss --routing falseIn order to help us style things up a little bit faster we will also use bootstrap and FontAwesome libraries.
Also, in order to navigate through the JSON response returned by the rest API, we will need to use json-query library.
NB: There are several libraries used for querying JSON objects. I chose json-query for its compact size and easy-to-use query syntax over other libraries such as jsonpath for example.
ng add @ng-bootstrap/ng-bootstrap
ng add @fortawesome/angular-fontawesome #choose free solid icons
ng add json-queryOnce it is set up, open the project with your favorite editor (VS Code in my case) and serve your application.
ng serve --port 4200Next, we will create our first component called dashboard-gallery. This component will contain the logic to make the calls to the rest APIs and sort the data accordingly. To support the logic, we will create a class called dashboard-card and an interface called dashboard-card-resource.
Along with those, we will also create a service called data.service. In a normal application this service would query an external endpoint with proper authorization but here we will simply mock this part and store all the data needed to perform the API calls in locally.
ng generate component dashboard-gallery
ng generate service data
ng generate class dashboard-card
ng generate interface dashboard-card-resourceNow, adding all the above to our app.module.ts, we should have something like:
CI providers rest APIs
Most of CI providers usually have a rest APIs in order to perform different types of operations without having to pass by the Web UI. In this tutorial we will take two of them: DevOps Azure and GitHub.
For DevOps Azure we will be looking to get a build pipeline/release result, while we will query the result of an Action run for GitHub. All and all, this is basically similar.
I will not go through the API itself and invite you to play with the different APIs if you have the time and look through the documentation if needed. (DevOps Azure and GitHub) 🧰
Authentication 🔑
DevOps Azure
As in any Rest API returning private data, you will first of all need to provide a token to your requests so the server knows that your are a legitimate user.
In DevOps Azure this token can be generated in this way:
Log in to your organization, click on user settings and select Personal access tokens
Click on New token
Give a name to your token, a custom expiration date (maximum one year) and the right scope. Only read access to build / release is needed. Finally press the Create button. If everything is ok you should see the following:
Make sure to store the token somewhere so we can use it in our application.
GitHub
Log in to your account and go to the settings. On the left side menu, go to the ‘Developer settings’ section and then go to the Personal access tokens.
On the menu, click on Generate new token, give “public_repo” access and finally click on Generate token.
Data needed to perform our API queries
Here are different JSON objects containing all the data needed to run our queries.
Each of them contain the following:
- absoluteUrl — One or several URLs that we need to query to retrieve the necessary data
- values — The JSON paths that will help us navigate the responses returned by the APIs and store the right data.
- variables — Some parameters to inject in the different queries so they return the right data.
From each of the resources we need we will retrieve the status and a direct link to navigate to it.
DevOps Azure build pipeline result
The query is quite straightforward and all the parameters needed as well.
DevOps Azure release pipeline stage result
This one is a little more tricky. We will need to make 3 different queries to get the result we need.
- The 1st query allow us to get our release id.
- In the 2nd we inject some data retrieved from the 1st query. in order to retrieve the stage id.
- In the 3rd we inject the data from the 1st and 2nd query to retrieve the execution result of our stage in the release pipeline.
NB: I am not really sure if there is a faster way to get the data we need, but as far as I remember this is what I I could find. ➰
GitHub action run
This is pretty straightforward with GitHub. All we need is to look for the first run in the list to get the latest result.
Dashboard component
Data
The JSON objects discussed above will be gathered in a single file data.ts, along with other type of data needed to run the requests properly (personal access tokens, name of the resource etc.), In turn this file will be used by our DataService class to make it available for our Dashboard class so it can use them to run the requests to the APIs.
Let’s now build our model, logic and view for the dashboard component.
Model
Let’s define a class for our Dashboard Card now.
Each card should have a set of different fields organized in different categories:
- First of all, some general information related to the resource itself, such as a family, name, datasource and a personal access token (pat).
- Then a set of variables that will be used in the queries. All these will be stored in an object called variables.
- Finally a set of resources such as the absolute url, and json queries to find the appropriate data in the response.
The link to the result and its status will also be stored on the object.
Service
As said earlier, all the data needed for our Dashboard cards will be stored in a single file called data.ts and served by a service called data.service.ts. The only thing it does here is to retrieve the different cards, and associate the right resource to each one of them before serving them to our component.
Component
Coming now to the logic of our dashboard component. We have different functions:
- A function called prior each API call that will substitute all parameters on our http request with the variables stored on the card object.
- A main function getCardData that will perform the http request to our APIs for each of our cards and store the results in the card object.
For resources that need multiple subsequent calls to retrieve the end results, we will call the method recursively while populating the http requests with the previous results.
- Each request is prepared before hand in order to add the authorization header to it.
Finally, as said mentioned earlier, after each API call we retrieve one or several values from the response via JSON query and store them in our object.
View
Now that we have all the data, the only thing we miss is our view. Each card should display the project name, the pipeline, and its status. Along with it we will also put a link that will lead us to the pipeline result, and a refresh button. That’s it.
NB: Two data pipes will also be needed to parse the data correctly: dashboard-status.pipe.ts and dashboard-style.pipe.ts but I will let you review them in the repo directly.
Here is how our view could look like:
Conclusion
Running the code in the repo with your own project and resources you should normally get something that looks like the following:
This example is rather simple, but depending on your need and context this can easily be adapted to other scenarios. 🤓
Deploying locally
If you would like to deploy your code locally without requiring a web server, simply make sure to remove the following line from your index.html:
...
<base href="/">
...and run the following command:
ng build --configuration productionThis creates a dist folder containing all the files needed to run your web application. You can open the index.html file in your browser, and it should work out of the box. If you move that folder somewhere else on your computer, just make sure to copy all the other files along with it (js, css etc.).
Code
All the code is available in my GitHub, feel free to re-use it as much as you need to, and don’t hesitate if you have any questions.
Going further
There are so many possibilities offered by each CI provider in their API and this example is only scratching the surface. Via the API, one can not only retrieve data about anything stored in your project (repo, builds, release, tests etc.) but also perform POST operation to create resources, trigger builds etc… With all this, one could create a powerful dashboard to quickly interact and monitor different projects.
Voilà, I hope that you enjoyed this post, and if you haven’t learned anything, maybe at least this gave you some ideas 😀
