Category: IBM

On using game theory to better understand and resolve DevOps issues (with help from LLMs)

I have seen a consistent set of issues when it comes to DevOps implementations.  The issues are:

  • Clients say they are emploing a DevOps lifecycle, but it’s more the case they are simply doing continuous integration and deployment (CI-CD) supported by an associated tool chain.
  • Many times the Dev (development) teams are separate from the Test teams and the Ops (operations/support) teams. Sometimes they are even in different silos in the organization. As a result, communication and collaboration — an critical part of a successful DevOps implementation — may be limited and weak.
  • As a result of that separation, the incentives may differ by team. In some cases they can end up almost competing against each other. (For example, I have seen Dev teams take time away from the Test teams in order to complete their assigned development tasks. The Devs “win” and the Testers “lose”.)
  • Quality standards held by the Test and Ops teams may not shift left to the Dev teams. As a result, quality responsibilities may lie mostly with Test/Ops, or there is a significant imbalance. (The Devs may do some limited unit or regression testing, but not enough to prevent bugs from being passed downstream in the DevOps lifecycle.)
  • Delivered functionality on time may become the primary driver for a project in order to meet the needs of key stakeholders, while the non-functional / quality of the system (e.g., performance, observability) may become secondary or even considered a Day Two problem.
  • As a result of all this, it’s not surprising to see Dev teams become focused and measured on delivering functionality at increasing velocities, while the Testers and to some degree the Operators have to maintain that focus / those measurements, while also being responsible for and measured on software quality.
  • In the end, you can get a situation where Devs are measured in terms of speed and functionality, while the Testers / Ops are measured on speed, functionality plus software quality. If delivering software is like a game, the Testers and the Operators are playing the game with a handicap due to quality metrics.

I have thought about these issues for some time and I wanted a framework to apply to them so I communicate in a more concrete way not only the issues but how to address them. I thought that one such framework to use could be supplied by game theory.

To apply game theory to DevOps, I started by considering the Devs as one set of players and Testers/Ops as another. To quickly work through some scenarios to see if such an approach was even valid, I decided to use LLM models. First I used GPT-OSS-20B to get my initial results. While it seemed promising, I noticed some errors creep into its analysis with regards to Nash equilibriums. I swapped out that model and used a larger LLM (provided by IBM Bob) to analyze and correct the initial results and produce a final analysis. You can read that final analysis, here.

I learned a few things from this exercise:

  • when the Devs are not bound by quality, a Nash Equilibrium is achieved, but it is one where they get a bigger payoff for going as fast as possible and the Testers/Ops have to settle for a situation where they suffer and get a worse payoff.
  • If the Devs are bound by quality, they get a lesser payoff but the Testers/Ops  get a better payoff and overall software quality can be increased. It seems obvious to do this, but working out a pay-off matrix helps with this.
  • Under certain conditions the Nash equilibrium can become weak for the Devs and that may result in them being less quality focused in order to succeed. Again, not surprising.
  • Game theory can show why Developers rationally can decide to forgo quality, but it may not be the best way to practically analyze your project. I’d simply recommend you continually work to spread quality measures throughout your DevOps lifecycle and to continually work to improve quality as the software matures.
  • LLMs can speed up the production of documentation just like it can speed up the production of code, but you don’t want to solely depend on it to get the documentation correct. Based on my understanding of game theory, I think its output is correct. If someone with a better understanding wants to correct me, I’d be happy to hear about it and I’ll be happy to amend this post.
  • For a more important project I would heavily modify the LLM documentation and take more ownership of it. This is more of a side project / exercise for me, so I decided to leave it as it is and move on to bigger things with bigger payoffs, to use the language of game theory. I hope you can still find it useful. I’d suspect the number of people who read it can be measured in one hand.

(P.S. The opinions expressed in this post are mine and do not necessarily reflect those of my employer.)

My dream of working from home started with this ad for IBM and Coppola’s Thinkpad (publish those visions you have)

The computer above, and the ad it is, came out in the mid 90s.

It was possible to work from home then, but it was not easy. I used to have a luggable computer that weighed 40 pounds and which I would …lug… home every day one summer to work from home. What I dreamed for, though, was to work from home with a small laptop like Coppola’s. A laptop where I could work from home daily, be it at a desk or in a beautiful kitchen like the one above.

It eventually happened. The laptops got better, the networks got better, and eventually the work cultures got better and I could do this. My kitchen wasn’t as nice, but everything else was nice.

Creative people, keep putting out your visions for a better world. You never know what dreams people will have. It might be as simple as a dream of working on a laptop in a kitchen. A dream that becomes more achievable once people can envision it.

What I find interesting in cloud tech, May 2023

It’s long past time to write about IT stuff I’ve been working on. So much so I’ve too much material to provide, and rather than make an endless post, I’ll focus on cloud. I’ve mostly been doing work on IBM cloud, but I have some good stuff on AWS and Azure (Sorry GCP, no love for you this time.)

IBM Cloud: most of the work I’ve been doing on IBM cloud has been hands on, as you can tell from these links:

Other clouds: Not so much hands on, but interesting.

The winter of smarter cities and why that’s a bad thing

In the early 90s, people soured on AI and work on it stopped. That period was known as AI Winter.

Smarter Cities is an idea likely in its own winter period. I concluded that when I watched the segment above on the subject. The segment’s focus (according to its Youtube comment)  was this:

In 2017, the City of Toronto embarked on a project with a subsidiary of Google called Sidewalk Labs. The idea was to develop a parcel of Toronto’s industrial waterfront in order to create a “smart city”. At first, the idea was met with a lot of enthusiasm, but eventually a number of concerned citizens, journalists and planners started to raise questions about data privacy, competition and public policy issues that Sidewalk Labs could not answer. What happened to Sidewalk Toronto and why are we so drawn to the idea of futuristic urban utopias? To help answer that: John Lorinc, author of “Dream States: Smart Cities, Technology and the Pursuit of Urban Utopias;” Josh O’Kane, author of “Sideways: The City Google Couldn’t Buy;” and Vass Bednar, Executive Director, Master of Public Policy in Digital Society Program at McMaster University.

It’s a good segment with good people critical of the idea of smart cities. Watching it, I could see why you might think that any city would be unwise to aspire to be a smarter one.

If you did think that, I’d ask you to think again. I believe cities, provinces, states and countries all benefit from becoming smarter. For example in Ontario, smart meters were deployed across the province to more accurately measure power consumption and help people shift their usage. Wastewater was measured during the pandemic to see if things are getting better or worse. And it’s not just public initiatives that matter: private services like Waze give drivers a view of the whole city and let them choose the best routes as they make their way to their destination.

Whenever there is municipal or state data available and software to process it, then you have a smarter city. It’s not at the scale that an organization like Google wanted it to be, but a smarter city nonetheless.

I strongly feel we need more of this. Smarter cities can be greener cities. Smarter cities can be better functioning cities. That’s why I am glad that Toronto is continuing to explore this, with things like its Digital Infrastructure Strategic Framework. And it’s not just Toronto: here is a list of the top 10 smart cities in the world and what makes them smarter.

It’s not just me that thinks this. Here’s Pete Buttigieg talking about smart city grants, and here is Bill Gates is talking about smart cities in Arizona.

Here’s to smarter cities. Smarter cities are better cities.

P.S. Wastewater examination is a smarter city activity, I think. Here’s more on wastewater surveillance for public health at Science.org. As for me, I check the province’s wastewater signal every week to keep track of COVID-19.

Also, some time ago I worked on the smart meter project for the province of Ontario. I led the infrastructure design and build for part of the overall system. Here is an IBM red paper I contributed to that talked about how to better design systems involved in smarter cities/planet projects.

IBM Cloud tip: take advantage of free IBM cloud products, including the IBM Kubernetes Service

IBM has numerous free products in its Cloud Service, and you can find them, here.

One I recommend especially is the Kubernetes Service. You can create a free cluster and learn a lot about both IBM Cloud and Kubernetes by using this.

If you aren’t sure where to start, I put together a github repo to help you get started. It gives you all the information you need, so you can go from a simple web page or node.js app on your own machine to having it up and running on the IBM Kubernetes service. You can find it here: blm849/networkcontainertesting: a simple way to test connectivity in and out of a container.

It’s up to date as of May, 2022. While there are plenty of tutorials out there, you may want to see if they are up to date. For example, some features may be deprecated.

Drop me a comment if you have any feedback. Good luck!

IBM Cloud tip: take advantage of tags to better manage your cloud environment

Last week I encouraged you to  consider your naming standards before adding services. This week I’d like to encourage you to use tags as well to help you manage your IBM Cloud environment.

 

As this piece from the IBM Cloud Docs on Working with tags explains, you can use tags to

organize, track usage costs, and even manage access to your resources. You can tag related resources and view them throughout your account by filtering by tags from your resource list. To see a full list of tags in your account, go to Manage > Account in the IBM Cloud® console, and select Tags. You can apply user tags to organize your resources and easily find them later or help you with identifying specific team usage or cost allocation. By creating access management tags, you can control access to your resources without requiring updates to your IAM policies.

Here’s some examples, partially taken from the same piece:

  • Use tags to identify or even manage access to your development environment, not to mention QA, UAT, Production and DR
  • Use tags to identify or even manage access to a project: project:lw-wizard, app:poc-app
  •  Use tags to define compliance requirements: dataresidency:germany, compliance:hipaa, compliance:pii
  •  Use tags to help you automate optimization: schedule:24×7, maxruntime:12days

So use tags: your IBM cloud environment will be easier to operate if you do.

For more on this, here’s a good blog post on tags, here: Characteristics of User and Access Tags on IBM Cloud | IBM

And this piece in IBM Cloud Docs is a good tutorial that will guide you:

…through the steps to centrally manage access to the resources in your account at scale (using tags). By completing this tutorial, you learn how to create an access management tag, add the tag to selected resources, and define a policy to assign access to resources based on the tags on those resources.

Good stuff. Start using tags more and you will find it much easier to manage your resources in the IBM Cloud. If you are unsure, start with a few for now (e.g. tag your production environment, tag resources belong to specific groups).  You’ll start to see the benefits soon.

On Pepper and Watson


If you have even a passing knowledge of IT, you likely have heard of Pepper and Watson. Pepper was a robot and Watson was an AI system that won at Jeopardy. Last week the Verge and the New York Times had articles on them both:

  1. Go read how Pepper was a very bad robot – The Verge
  2. What Ever Happened to IBM’s Watson? – The New York Times

I don’t have any specific insights or conclusions into either technology, other than trite summations like “cutting edge technology is hard” and “don’t believe the hype”. AI and robotics are especially hard, so the risks are high and the chances of failure are high. That comes across in these two pieces.

Companies from Tesla to Boston Dynamics and more are making grand claims about their AI and their robotics. I suspect much of it will suffer the same fate as Pepper and Watson. Like all failure, none of it is final or fatal. People learn from their mistakes and move on to make better things. AI and robotics will continue to advance…just not at the pace many would like it too.

In the meantime, go read those articles.  Especially if you are finding yourself falling for the hype.

(Image: link of image on The Verge)

How to get up to speed really quickly on Kubernetes and Docker if you are an infrastructure specialist

If you are an infrastructure person and you are trying to ramp up really quickly on Docker and Kubernetes, here are some good links to get you started:

I also have this repo on github that can help.

Is this the last word? Good lord, no. But it can help you stay in the conversation and helps you map all this stuff to networks and processes and files and VMs and services and other tech you are used to.

You’ve heard of Google’s 8.8.8.8…now there’s 9.9.9.9 (a new DNS service from Quad9/IBM)

If you are using Google’s DNS services (i.e., you are using 8.8.8.8 for DNS services), then there is a new provider on the block you should consider: Quad9. The following links detail why you might want it, as well as how to set it up.

We need all the help we can get in dealing with malicious people and computers on the Internet. Glad to see my employer has a role in this.

Git 201: how to get to the next level with git


There are quite a few really good introductions to git. I’ve written about them here. Once you get past “git 101”, where do you go to learn more and be more productive with git? I’d like to recommend this article:
developerWorks: Learn the workings of Git, not just the commands. It should help you get to the next level. I particularly like the diagrams: there are alot of them, and they help you better understand the flow that can occur when you really start capitalizing on git.