Tag Archives: AWS

What I find interesting in cloud tech, May 2023

It’s long past time to write about IT stuff I’ve been working on. So much so I’ve too much material to provide, and rather than make an endless post, I’ll focus on cloud. I’ve mostly been doing work on IBM cloud, but I have some good stuff on AWS and Azure (Sorry GCP, no love for you this time.)

IBM Cloud: most of the work I’ve been doing on IBM cloud has been hands on, as you can tell from these links:

Other clouds: Not so much hands on, but interesting.

Advertisement

My notes on falling to build a Mastodon server in AWS (they might help you)

Introduction: I have tried three times to set up a Mastodon server and failed. Despite abandoning this project, I thought I would do a write up since some people might benefit from my failure.

Background: during the recent commotion with Twitter, there was a general movement of people to Mastodon. During this movement, a number of people said they didn’t have a Mastodon server to move to. I didn’t either. When I read that Dan Sinker built his own, I thought I’d try that too. I’ve built many servers on multiple cloud environments and installed complex software in these environments. I figured it was doable.

Documentation: I had two main sources of documentation to help me do this:
Doc 1: docs.joinmastodon.org/admin/install/
Doc 2 gist.github.com/johnspurlockskymethod/da09c82e2ed8fabd5f5e164d942ce37c

Doc 1 is the official Mastodon documentation on how to build your own server. Doc 2 is a guide to installing a minimal Mastodon server on Amazon EC2.

Attempt #1: I followed Doc 2 since I was building it on an EC2 instance. I did not do the AWS pre-reqs advised other than create the security groups since I was using Mailgun for smtp and my domain was elsewhere at namecheap.

I did launch an minimal Ubuntu 22.x server that was a t2.micro, I think (1 vCPU, 1 GiB of memory). It was in the free tier. I did create a swap disk.

I ran into a number of problems during this install. Some of the problems I ran into had to do with versions of the software that were backlevelled compared to doc 1 (e.g. Ruby). Also I found that I could not even get the server to start, likely because there just is not enough memory, even with the swap space. I should have entered “sudo -I” from the start, rather than putting sudo in from of the commands. Doing that in future attempts made things easier. Finally, I deleted the EC2 instance.

Attempt #2: I decided to do a clean install on a new instance. I launched a new EC2 instance than was not free and had 2 vCPU and 2 GiB of memory. I also used doc 1 and referred to doc 2 as a guide. This time I got further. Part of the Mastodon server came up, but I did not get the entire interface. When I checked the server logs (using: journalctl -xf -u mastodon-*) I could see error messages, but despite searching for them, I couldn’t see anything conclusive. I deleted this EC2 instance also.

Attempt #3: I wanted to see if my problems in the previous attempts were due to capacity limitations. I created a third EC2 instance that had 4 vCPU and 8 GiB of memory. This installation went fast and clean. However despite that, I had the same type of errors as the second attempt. At this point I deleted this third instance and quit.

Possible causes of the problem(s) and ways to determine that and resolve them:
– Attempt the installation process on a VM/instance on another cloud provider (Google Cloud, Azure, IBM Cloud). If the problem resolves, the cause could be something to do with AWS.
– Attempt this on a server running Ubuntu 20.04 or Debian 11, either on the cloud or a physical machine. If this resolves, it could be a problem with the version of Ubuntu I was running (22.x): that was the only image available to me on AWS.
– Attempt it using the Docker image version, either on my desktop or in the cloud.
– Attempt to run it on a much bigger instance. Perhaps even a 4 x 8 machine is not sufficient.
– See if the problem is due to my domain being hosted elsewhere in combination with an elastic IP address by trying to use a domain hosted on AWS.

Summary: There are other things I could do to resolve my problems and get the server to work, but in terms of economics: the Law of Diminishing Returns has set in, there are opportunity costs to consider, the sunk costs are what they are, and the marginal utility remaining for me is 0. I learned a lot from this, but even if I got it working, I don’t want to run a Mastodon server long term, nor do I want to pay AWS for the privilege. Furthermore, I don’t want to spend time learning more about Ruby, which I think is where the problem may originate. It’s time for me to spend my precious time on technologies that are personally and professionally better rewarding.

Lessons Learned: What did I learned from this?

– Mastodon is a complicated beast. Anyone installing it must have an excellent understanding of Linux/Unix. If you want to install it on AWS for free, you really must be knowledgeable. Not only that, it consists of not only its own software, but nginx, Postgres, Redis and Ruby. Plus you need to be comfortable setting up SSL. If everything goes according to the doc, you are golden. If not, you really need an array of deep skills to solve any issues you have.

– Stick with the official documentation when it comes to installing Mastodon. Most of the many other pages I reviewed were out of date or glossed over things of note.

– Have all the information you need at hand. I did not have my Mailgun information available for the first attempt. Having it available for the second attempt helped.

– The certbot process in the official document did not work for me. I did this instead:
1) systemctl stop nginx.service
2) certbot certonly –standalone -d example.com (I used my own domain and my personal email and replied Y to other prompts.)
3)  systemctl restart nginx.service

– Make sure you have port 80 open: you need it for certbot. I did not initially for attempt 3 and that caused me problems. I needed to adjust my security group. (Hey, there are a lot of steps: you too will likely mess up on one or two. :))

– As I mentioned earlier, go from the beginning with: sudo -i

– Make sure the domain you set up points to your EC2 instance. Mine did not initially.

Finally: good luck with your installation. I hope it goes well.

P.S. In the past I would have persevered, because like a lot of technical people, I think: what will people think of me if I can’t get this to work?? Maybe they think I am no good??? 🙂 It seems silly, but plenty of technical people are motivated that way. I am still somewhat motivated that way. But pouring more time in this is like pouring more money into an old car you know you should just give up on vs continuing to try and fix.

P.S.S. Here’s a bunch of Mastodon links that you may find helpful:
http://www.nginx.com/blog/using-free-ssltls-certificates-from-lets-encrypt-with-nginx/
app.mailgun.com/app/sending/domains/sandbox069aadff8bc44202bbf74f02ff947b5f.mailgun.org
gist.github.com/AndrewKvalheim/a91c4a4624d341fe2faba28520ed2169
mstdn.ca/public/local
http://www.howtoforge.com/how-to-install-mastodon-social-network-on-ubuntu-22-04/
http://www.followchain.org/create-mastodon-server/
github.com/mastodon/mastodon/issues/10926

Kubernetes and Clouds and much more (What I find interesting in tech, July 2022)


Since April, here are a ton of links I found useful while doing my work. Lots of good stuff on Kubernetes and Cloud (both IBM’s and AWS’s); some cool hardware links; some worthwhile software links. Plus other things! Check it out.

Kubernetes: plenty of good things here to explore if you are doing things with Kubernetes like I was:

Terraform: Relatedly, I was doing work with Terraform and these were useful:

IBM Cloud: one of the two clouds I have been working with. Alot of the work was Kubernetes on IBM Cloud so you’ll see some overlap:

AWS: I work on alot of cloud providers. Mostly IBM Cloud but others like AWS

Software: some of these were work related, but some are more hobby oriented.

Hardware: the pickings are few here

Finally: here are an odd assortment of things worthwhile:

What I find interesting in tech in general, Apr 2022


Well, I didn’t expect this week to be “what I find interesting in IT Week” , but here we are! I hope you have found it all useful. While the other posts were specific, these are of a general nature. From weird cool stuff to mainframes to iphones to architecture. Dig in!

IT Architecture: Here’s some tools you IT architects can use:

Cloud: some cloud related stuff:

Programming: here’s some things software and hardware folks might find interesting:

Finally: here’s some links that need to be seen without falling into a particular category:

IBM Cloud, Terraform and DB2 (or some of what I find interesting in tech, nov 2021)

Wow! I haven’t done one of these posts on things I have found interesting in tech since July! So of course there is a lot here! I need to do these more often.

For this one it is mostly on the cloud and IBM cloud especially. Not so much Kubernetes this time, but lots on Terraform and DB2 in the cloud. A bit of IoT. Some software. I have a section on IT history which I like too.

Grab a coffee or tea or what have you and dig in. Take what you can use.

IBM Cloud: As usual, I’ve been doing work on cloud…mainly IBM Cloud. Here’s some IBM Cloud Docs on using their API. This on the IBM CLI is  a good reference. Here’s a good Alerts Overview on LOGDNA. This is massively helpful: the API Reference for  https : //sldn.softlayer.com/ … it’s very useful on how to use the API to work with IBM cloud. Here’s something on Alerts using sysdig. More on tracking:  IBM Cloud Monitoring Logging and Activity Tracker with Teams ( a good repo).

Other clouds: While I support IBM Cloud, if you are leaning otherwise, this could be helpful: Accelerating your Migration to AWS. Speaking of  AWS: Augmenting VMware Cloud on AWS Workloads with Native AWS services. Here are some pieces on Azure: Microsoft Azure cloud vulnerability is the worst you can imagine. I have no comment. Hey, maybe it’s time to get off Windows and SQL Server 2012 (or run them on Azure). If that’s you, read that.

On cloud in general, in case you were wondering, the answer to this question: Resiliency Is Automatic When I Move to the Cloud Right? is No. Here’s an interesting piece: The love/hate relationship the cloud has with Linux

Time flies! Happy 15th Birthday Amazon EC2. Lastly, here are the 5 Biggest Cloud Computing Trends In 2022 

Terraform: in working on cloud recently, I’ve been using Terraform and gathering links on using it. Lots of them. Here they are in somewhat random order. For example, discover best-practice VPC configuration for application deployment. Another piece on   IBM and Terraform.  Here’s more on it. Need a terraform Template for Monitoring with Sysdig Teams? If you need to plan create and update deployment environments using TF and IBM Cloud. This is a good blog post on Provisioning IBM Cloud Services With Terraform. If you need to deploy a n-Tier Web App in a Virtual Private Cloud using Terraform & Ansible. This piece is essential if you want to create services in IBM Cloud using Terraform IBM Cloud Services Info. Here’s how to give a .tf file as input in Terraform Apply command.  A page on data sources in Terraform resources explained with example. How about how to create Multiple Instances in a VPC Using Terraform. Or how to create reusable infrastructure with Terraform modules. Or a VPC. Or an n-Tier Web App in a Virtual Private Cloud using Terraform & Ansible .Here’s a piece on IBM Cloud Toolchain- Managed Devops for Schematics/Terraform. If you want to create Virtual servers in IBM cloud Terraform with for loop. A good intro: Getting started with IBM Terraform provider.

Still more on getting started with Terraform. Something harder: How to deploy high-availability web app using Terraform.

DB2: I have been doing lots of work on using DB2 on IBM Cloud. Here’s something on querying the IBM Cloud Databases API from the Command Line. Here’s something on using RESTful APIs and Microservices to Work With Db2. Here’s some examples of using the DB2 API: DB2 get about info,  and  Db2 get overall stuff, and Using the DB2 API.

Kubernetes: doing less stuff on Kubernetes this month, but I thought these were good: OpenShift vs. Kubernetes: What is the Difference? Helpful: Enable Rolling updates in Kubernetes with Zero downtime. Also helpful: Configure Liveness Readiness and Startup Probes.

Software (plus AI): for my DB2 work, I was calling the APIs using bash scripts. Here’s the answer to the question: What is the simplest way to dockerize a bash script? from Quora . Here’s something on executing a SHELL script in a docker container. Relatedly, here’s how to run a Bash script in an Alpine Docker container? 

Here’s some Unix pieces: Canonical extends Ubuntu 14.04 and 16.04 life cycle to 10 years. Good! Here’s how to do this on Debian 9:upgrade python 3.5 to python 3.7. Relatedly, here’s how to Install Python 3 on Mac – Brew Install Update Tutorial. Also related: Python on MacOS (Big Sur) . More related stuff on this:  How to fix “macOS: xcrun: error: invalid active developer path missing xcrun” error? 

I used to love the language APL. Here are two things on it: Is APL Dead? and Dyalog APL Tutor. I am curious to see how this plays out: Microsoft announces Windows 11 SE a new Chrome OS competitor.

Here’s two random AI links: The Secret Bias Hidden in Mortgage-Approval Algorithms (bad) and Motorist fined after CCTV confuses his number plate with woman’s T-shirt (funny bad).

IOT: Need to build front panels for your IOT projects? This is good for them. Relatedly the Ultratroninator3000.

Speaking of IOT projects, here’s some worth checking out: Top project ideas for the Raspberry Pi Zero. Then there is this: Simple Raspberry Pi Weather Station. Related: E-Ink Tide and Weather Tracker. This is a cool project for finance folks:  This tiny IoT ticker-meter turns your tabletop into a miniature stock forex and crypto market! Nice: Kobowriter transforms the Kobo Glo HD into an E Ink typewriter. I loved this: The Simpsons TV Made IRL with Raspberry Pi @Raspberry_Pi #PiDay #RaspberryPi.

IT History: here are a few good pieces on IT history. This was a great piece on how the iPod was developed: A Prototype Original iPod. Going back in time, here’s a good article on Sinclair’s amazing 1974 calculator hack. There was lots of talk about Sinclair computers after the great Home computing pioneer Sir Clive Sinclair dies aged recently at 81. Meanwhile in ancient history, IT wise: 50 years ago today the first UNIX Programmers Manual is released.

IBM History: I was reading this piece on  How IBM “lost the cloud” and it got me thinking. First, I don’t think IBM has lost the cloud business. I also think that IBM’s history is never straightforward and it is risky to count it out. For example, after e-business, IBM tried to promote the idea of autonomous computing. Here’s two pieces on it: Autonomic computing and Q&A: IBM sticks to autonomic computing agenda. It was a good idea, and it supported the work IBM was doing in the Tivoli space, but it was not as big a success as e-business IMHO. From there IBM did work on their Smarter Planet campaign and I believe it was more successful. I did some work in this area myself. From there IBM went into cloud. First there was a homegrown service, and then IBM bought Softlayer and went with that to compete with Amazon and then later Microsoft and Azure. For now the history is still being written. No one has won or lost until cloud is over or someone exits the field. Again, my opinion only.

Cool stuff: here’s something on a A LOST 1981 TRS-80 ADVENTURE GAME (SLIGHTLY REMASTERED FOR THIS CENTURY). Do you want to Turn your Android phone into a pocket Mac Plus? .Of course you do! Here’s a cool tool:  Doodle Ipsum. Check this out: This tiny Simpsons TV lets you watch tiny Simpsons TV. Very fun!

Here’s some cool Microsoft stuff:  Microsoft accounts to no longer need passwords and how to use a VBA procedure that deletes the current page in a Word document. Also this: Office Editing for Docs Sheets & Slides.

Here’s some cool Mac stuff: 12 Clever Apple TV 4K Settings Everyone Should Know About and macOS Terminal commands every Mac user should know

Very cool:  Need a new monitor for your computer? You can wear one on your face.

Generally:  here’s how to How to Mass Delete Emails in Gmail. Here’s a PrinCube Mobile Printer. I like it. Another cool device is this gloriously Fixable Laptop.

Speaking of laptops, here is the  Best cheap laptop of 2021. And here is something else cheap: The HP Chromebook 14 is just over $230 at Amazon right now. Relatedly, here’s 9 Reasons You Should Buy a Pixel 5a Over the iPhone SE.

These are some simple free fully responsive one-page sites for pretty much anything. And Elon Musk says Tesla is working on humanoid robots…sure…whatever edgelord.

Finally: document your code.  And remember, no matter how fast your networks get….

… never, never, never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. 🙂

As always: thanks for reading this!

(Top photo from the story on the ipod. Bottom photo by Mostafa tarek on Unsplash)

 

Need a temporary Mac? Now with AWS, you can get one!

How? By using their cloud service: AWS brings the Mac mini to its cloud.

Perfect for those times you need access to a Mac for a short period of time (e.g. testing software).

Throughout my career I have been involved with Macs and cloud technology. I remember when Apple made Mac servers. There was even a separate MacOS for them. So I am loving this evolution and the repositioning of Macs in a data center.

Image from here which also has a write up on this.

 

One way to backup your files is to the cloud


I haven’t tried this yet, but I am seriously considering it. I already use AWS for other things, so it might be time to see if I can archive old files that I rarely use but don’t want to delete. I don’t feel like getting more hardware, and I don’t have much confidence in Apple’s iCloud. So this might be the solution: ​How to Use Amazon Glacier as a Dirt Cheap Backup Solution.

And if I go that route, I will need tools. This article should give me the information  I need for that: The Best Tools for Uploading Files to Amazon Glacier – Digital Inspiration.

(Photo by Glen Carrie on Unsplash)

Quote

One of the the better reviews of WeWork and their IPO is naturally this one, by Stratechery


Stratechery is always great and this piece is no different: The WeWork IPO – Stratechery by Ben Thompson.

What makes it good is that rather than just slamming WeWork superficially, as many takes have, it delves into what could possibly justify why WeWork is a good investment.

My take is that if WeWork had a different executive, it could be a successful company. I think the comparison to AWS is somewhat valid, and in the gig economy with lots of short term work, it could become very successful. (It worked really well for a recent project I was on).

That said, I believe the executive team of WeWork will not be able to handle any drying up of capital or a recession of any length. Or investors will wake up and ask themselves why WeWork should be valued way more than IWG/Regus. Time will tell, of course.

One last thing: my understanding is that WeWork had to start from scratch in terms of buying up / leasing real estate, but AWS did not start from scratch and took advantage of existing capacity Amazon currently had.

(Image link to the original piece in the article reference)

Quote

How to set up a static website for about $1 a month

Finally got around to doing this, so I have update this page. Here’s your options. Option 3 worked for me.

Note, you will need some technical knowledge, an Amazon account, and one of these guides:

  1.   Simple site hosting with Amazon S3 and HTTPS. This is good if you plan to host your DNS at AWS. I didn’t, so I ran into trouble.
  2. . This piece is similar and helped too but the SSL certificate was hosted elsewhere, so it only helped so much.
  3. This page finally helped me because I host with namecheap.com. I followed it and it worked great! It has more detail than some of the others.

You don’t have to use AWS. Other cloud providers offers something similar:

So, lots of options.

Quote

How I came up with the web page: All the books I have read since 2017 (somewhat technical. Involves python, S3 buckets)

 


I used to be a haphazard reader and my reading had slacked off. In 2017 I decided to have a goal of reading more and recording the books I had read. For the record, I had a simple Excel spreadsheet. This was good, but not easy to share.

 

To build this page, All the books I have read since 2017 | Smart People I Know, I wrote a Python program to convert the Excel spreadsheet to HTML. After that, it make it look modestly better, I stole some ideas from here. I was going to put the HTML directly into WordPress, but there were formatting issues. I instead put the page in an S3 bucket at AWS. And voila! Done!

 

If you are cleaning up an environment by deleting resources in Amazon’s EC2, here is a checklist to get you started

I just cleaned up an environment I had set up in Amazon years ago for a client. (The client wanted to use Amazon, so we did.) In doing so, I wanted to make sure I didn’t leave anything behind which would cause me to continue getting billed even though I was no longer actively using EC2. I believe that the following checklist was useful in insuring this.

My EC2 cleanup checkist:

  1. Delete my Elastic IPs
  2. Terminated instances – running and non-running  (I did this before deleting volumes, since it deleted alot of them for me)
  3. Delete remaining volumes
  4. Delete my security groups ( 1 will be left – the default one)
  5. Deregister AMIs
  6. Delete snapshots (you need to deregister your AMIs before you do this)
  7. Check your account balance
  8. In a few days, check your account balance to see if there are any charges you haven’t accounted for

After following this checklist, my EC2 environment was cleaned up. Depending on how you are using EC2, you may have more things to delete. Checking your account balance will help there: if you left things behind, they may incur charges. An increase in your account balance will help flush them out.

One thing to consider: you may delete something, but it doesn’t show in admin console. If that is the case, logout and then in. I did that when I was having trouble deregistering my AMIs. I logged out and then in and when I checked them, they were now deregistered.