Back to the Sky: Processing Satellite Data Using Cloud Computing

From time to time, I work with researchers on projects outside of the day to day, forms-over-databases stuff. Mind you, I like a nice form over a cool data repository as much as the next guy, but it’s definitely cool to stretch your arms and do something more.

So, when Dr. Ogundipe approached me about cloudifying her satellite data processing, I had to do a lot of research. She had a processing pipeline that featured ingesting satellite images, doing some data cleanup, then analyzing those results using machine learning. Everything ran on local machines in her lab, and she knew she would run into scaling problems.

Early on, I decided to use a containerized approach to some of the scripting that was performed. The python scripts were meant to run in Windows, but I had an easier go at the time getting Linux containers up and running, so I went with that. Once the container was in good order, I stored the image in the Azure Container Registry and then fired it up using an Azure Container Instance.

Like a good story, I had started in the middle – with the data processing. I didn’t know how I would actually get non-test data into the container. Eventually, I settled on using Azure Files. Dr. Ogundipe would upload the satellite images via a network drive mapped to storage in the cloud. Since I got to have some fun with the fluent SDK in Azure a while back, I used it to build an orchestrator of sorts.

Once the orchestrator had run, it would have fed the satellite images into the container. Output from the container was used to run models stored in Azure ML. Instead of detailing all the steps, this helpful diagram explains the process well:

Super simple.

No, not that diagram.

The various cloud resources used to process satellite data in Azure.

So, I shared some of this process at a seminar Dr. Ogundipe held to talk about the work she does, and how her company, Global Geo-Intelligence Solutions Ltd uses a pipeline like this to detect locust movement in Kenya or the impact of natural disasters and a host of other applications of the data available from satellite images.

status.[utility].co.tt

Recently, there was thunder and lightning and heavy rainfall in Trinidad. Trees fell, and there was a range of property damage.

In cases like this, the aftermath can involve service outages across a number of service providers. For me, it was power. There was a bright spark, a loud bang and then silence.

Right after the first jumble of thoughts, I remembered you can call the power company to find out about outages. In this case, it was 800-BULB. The operator was prompt, he helpfully indicated that it was “a big problem that will take a long time to fix”.

When we awoke hours later, there was still no power, but added to that, I could not reach the power company on the phone. Either a lot of ringing or busy signals. And it made me think, calling to find out about the status of service is an approach we should retire.

From a cloud perspective, we can readily find out if a given provider is up or down by checking one of their status pages. Both Azure and GitHub comes to mind.

If we had access to the numbers, I expect the power company’s lines didn’t stop ringing until the day after the event. How did they handle the increased load and calls? That could have been lessened if there was an automated way their customers could get answers. Both at the phone level, and on the web.

I expect that some of the more advanced utility companies here in Trinidad already have internal systems that can readily provide this information, it’s therefore a matter of going the last mile and making it accessible to the public.

Of course, if utilities use a standard mechanism for reporting, then what can then happen is the status of a number of public utilities can be known to the nation, without having to make any calls.

What a day that would be.

Build 2020: Top 5

So, Microsoft’s Build 2020 Conference was held a few weeks ago. And at Teleios, we normally try to check it out because you get a good sense of where they’re going with some of the technologies and platforms we use to build solutions.

This year’s effort didn’t fail to deliver, even though the format had to be drastically change because of the strict rules of social distancing most of the planet was under due to COVID-19.

On a run a few days ago, I heard Jeff Sandquist, CVP dev-rel at MSFT talking about what it took to put on the event, likening it to a production by a movie studio. It was a great interview on a podcast called Screaming in the Cloud, in which Sandquist shared his top 5 Build Announcements (starts at 41:40):

1. Microsoft Teams: See it as more than a communication channel; it’s a development platform.
   “Single largest development opportunity for developers across the planet”
2. Developer Productivity: A number of changes and new resources have been announced to make it easier, faster and more productive when developing solutions.
3. Power Apps: Continuing commitment to enabling citizen developers to build no-code solutions that frees up system and other software developers to focus on other concerns while the organization gets things done. There was a lot to see about Power Apps and Power Platform and build.
4. “Making the best damn developer box” – Scott Hanselman’s keynote highlighted many new improvements to windows itself and the development experience there that underscore Microsoft’s goals around improving the tools that support the development process. From the new Windows Terminal, to making 2020 (finally), the year of Linux on the Desktop.
5. Microsoft Learn. Learning how to get into Azure and other Microsoft development technologies can be a challenge, to put it lightly. But Microsoft has recognized this and deliberately made learning about it’s products a strategic asset. From easy on-ramps via videos on the new Microsoft learn site, to very low barrier-to-entry access to the cloud through Azure, Learn its where it’s at. More seasoned developers will be happy to hear too, that documentation is seen as a key part of the developer journey and this is reflected in the way they approach it.

It was helpful to hear Sandquist talk about Microsoft’s Founding Moment. That they started as a Developer First company and they’re staying true to their roots.

Caribbean Developers’ Salary Survey – Redux

I saw on the Caribbean Developers’ Facebook group someone looking to run a salary survey:

I dig Matthew’s approach, because instead of doing the survey and trying to see if there’s interest, he’s looking to see if there’s any interest first and then leaning into the work.

From the looks of it, interest is low, maybe it will grow, but I wouldn’t be surprised if it doesn’t.

In 2018, I had a similar idea, for the same Caribbean Devs group. I may have been inspired by the StackOverflow Developer Survey and my own curiosity as it relates to the job market.

Here’s my survey results from back then:

caribbean-developer-salary-survey-2017Download

Just about 40 submissions were made, which was instructive. I appreciated that if you do a salary survey in a small pool, it can be too revealing if you have access to each submission, even if the submissions are anonymized. If I were to do this again, I would bear that in mind, I might leave out a Company field, for example, in favor of Industry. I’d also not include a Job Title field and instead leave it at Category and perhaps Years in the industry. I might also include questions that get to the heart of project diversity, too.

So, this is mostly a cautionary tale about surveys like this. There are good points of guidance to get from economists and people in the social sciences to understand how to design things like this, beyond simply capturing the data.

Nurse Carter – a COVID19 hackathon winner

So, when I built Nurse Carter years ago, I got the data from the Ministry of Health’s website in Trinidad and Tobago. I hadn’t updated the data since. Also, I built it with version 3 of the Microsoft Bot Framework.

Bot Framework updated a lot since then, and so did the health facility schedule. I took the opportunity to hit two birds with one code. I updated to the latest bot framework bits, changed from Node to dotnet and updated the health facility data.

To make it COVID-19 relevant, I included information about that as a menu option in Nurse Carter.

When I first did Nurse Carter, I feel like I had a lot more time – lol, I did because I was on “paternity leave” (it’s regular vacation I’m calling that). Now, in snatches of time, I found myself thinking “this is rell work!”

Nevertheless, I got to the submission line and Nurse Carter to a devpost hackathon and it was one of the top projects out of 1500 that were submitted! COVID-19 has brought a lot of heartache but I’m hoping that innovative approaches like Nurse Carter get more and more opportunities to deliver meaningful value in people’s lives.

That time when global quarantine liberated digital services. #CoronaEffect

There has been a sudden, dramatic step-wise improvement in digital services in Trinidad & Tobago.

• 

• 

• 

You can place an order via WhatsApp for groceries in Trinidad & Tobago now. From one of the largest chains in the country.

Many government agencies are available in a real-time way on social media. Providing updates of a COVID and even non-COVID nature

A local bank has started to enable sending of money that the recipient has access to in 1 day, down from 4 to 5. The others, I expect are sure to follow.

I’m here for all of it. Maybe this activity is just merchants and other service providers responding to the change in environment. Everyone’s online, and the only way to get people to interact meaningfully with your service is by meeting them where they are at.

For now, I’ll list these services as I hear about them. I won’t celebrate too loudly, because, you know, many people have been clamoring for this for a long time.

Parse a pdf, quickly

1. You have docker installed.
2. You don’t have time to futz around with PDFs

If you have a PDF you’d like to get tables out of, tabula works a dream. When last I used it successfully, I was on my mac. I jumped on to a windows machine, and I just couldn’t get it to run.

*a few googles later*

I found a docker command that hosts a web API that gives access to tabula:

docker run -p 8080:8080 -e HOST=0.0.0.0 gavinkflam/tabula-api:1.0.0

At the readme for this container, the example was this:

curl -X POST -H 'Content-Type: multipart/form-data' \
  -F 'file=@my-pdf-file.pdf' -F 'guess=true' -F 'pages=all' \
  http://localhost:8080/api/extract

Thankfully, because of WSL, I just popped over into bash and I had tables in a flash 😎

I’m a student, where can I Azure? (freeness welcome)

Whenever I teach cloud technologies, I emphasize practical applications, even for the newest students. This really helps the topic come alive and allows the students to engage the material in a more realistic way.

As it turns out, a few of my friends reached out to me about Azure resources for learners, so, I thought I’d just jot down a few resources here.

Azure for students

So, a simple search online should point you here:

https://azure.microsoft.com/en-us/free/students/

One of the first questions that come up is “Am I worthy eligible?”, which is helpfully answered in the FAQ:

Azure for faculty

Now, if you’re an educator, teaching assistant or other faculty, you can sign up for open source content – I’ve used this, it’s legit and the github repo is very active! As well as get a free Azure account for doing demos and showing how the stuff works.

Just plain old free Azure stuff

If you aren’t enrolled anywhere and just want to learn Azure stuff, there are also a lot of free-levels that you can use to get started.

https://azure.microsoft.com/en-us/free/

When you access this option, they’ll ask for a credit card number – this is for verification, they won’t charge you automatically unless you choose to upgrade. You have to choose to upgrade, it’s not going to sneak up on you.

Some azure services have levels that are always free, like Azure App Services, Azure Functions and Azure Kubernetes Service. There are others that are free for the first 12 months of usage. You can check out the full list here (scroll down).

So, there you go, get out there, get started, have fun!

Trinidad and Tobago Azure Users Group

We’re having the first of many (I hope) Microsoft Azure users meeting this Thursday (February 13).

How do you Azure?

I’m excited by this because here in Trinidad, I’m grateful that in and around Teleios, we’ve been using Azure for years, and I’ve often wondered about who else around us is learning, playing and building on the platform. Nigel Sammy, Christian Anton and I will be sharing on Azure in general, Azure DevOps and Azure Arc. It’ll be a really nice spread of features, from the novel to the staid.

So, convince you’re boss, if you’re interested, we’ll be at the Microsoft Office on Mucurapo Road, from 9:00 to 11:00 am.

The event is free, so’s the breakfast and you can register here.

Three things about January

Just before I went back out to work in January, I had a fair sense of how I wanted to start the year. At work, I’d be diving in to Planner and refining some early 2020 goals that Anand and I had come up with. At projects, I’d be doing some arduino and bot stuff. All great.

But from January 2nd, I found myself very deep in legacy code from at least a decade ago. And I was working very late. It’s like time stood still. When my team and I raised our heads for a week, a change to the environment around that same code meant even more speelunking.

Out of this last few weeks I saw three things of note:

try
{
//actual code to do something meaningful
}
catch
{
//empty, meaningless catch block
}

1. Stop using empty catch blocks.
   I haven’t had cause to leave dangly, useless catch blocks lying around for a few years, but I remembered when I used to. And boy was it a shocker. In the age of amazing tracing tools like App Insights, throwing away trace opportunities was like a slap in the face.
   When I showed a few developers at work some examples of empty catch blocks, they understood why it might have happened – I didn’t know what I wanted to do with the exception at the time – but my response was largely, “well, at least log it na”. At least. Even logging took a hit this month.
   
2. Log with sense
   The worst kinds of exception message in a log is “An exception occurred”. For the most part, sometimes, when looking for the flow of a problem, I might just put a log statement to show that we got to a place in code. Sometimes, that place is an error handler. The way I see exceptions now, if you’re logging that it occurred, take some time to log what that block was trying to do, as well as what parameters are available. In a messaging system for example, simply a message Id is a big deal. So, first we log, then we log with sense.
3. Sleep is very good.
   Not just sleep, but a firm commitment to not rush off with the first good idea you have when facing a problem. I have actually heard this stated another way, “Beware the danger of the first light”. While trying to debug our way out of the hole we were in, several ideas came up. Most recently, I had a great one that would have led to massive changes in the system. I was about to announce it, but one of the managers on our team cautioned me. She said it was Friday, sleep on it and see if you feel the same way on Monday.
   Monday came, I felt the same way. Until I spoke to one of the newest engineers on our team and realized we could achieve what I wanted to with a change to one module, as opposed to the whole system. That was good.
   Over the course of this month, there were a lot of challenges.