Back to the Sky: Processing Satellite Data Using Cloud Computing

From time to time, I work with researchers on projects outside of the day to day, forms-over-databases stuff. Mind you, I like a nice form over a cool data repository as much as the next guy, but it’s definitely cool to stretch your arms and do something more.

So, when Dr. Ogundipe approached me about cloudifying her satellite data processing, I had to do a lot of research. She had a processing pipeline that featured ingesting satellite images, doing some data cleanup, then analyzing those results using machine learning. Everything ran on local machines in her lab, and she knew she would run into scaling problems.

Early on, I decided to use a containerized approach to some of the scripting that was performed. The python scripts were meant to run in Windows, but I had an easier go at the time getting Linux containers up and running, so I went with that. Once the container was in good order, I stored the image in the Azure Container Registry and then fired it up using an Azure Container Instance.

Like a good story, I had started in the middle – with the data processing. I didn’t know how I would actually get non-test data into the container. Eventually, I settled on using Azure Files. Dr. Ogundipe would upload the satellite images via a network drive mapped to storage in the cloud. Since I got to have some fun with the fluent SDK in Azure a while back, I used it to build an orchestrator of sorts.

Once the orchestrator had run, it would have fed the satellite images into the container. Output from the container was used to run models stored in Azure ML. Instead of detailing all the steps, this helpful diagram explains the process well:

Super simple.

No, not that diagram.

The various cloud resources used to process satellite data in Azure.

So, I shared some of this process at a seminar Dr. Ogundipe held to talk about the work she does, and how her company, Global Geo-Intelligence Solutions Ltd uses a pipeline like this to detect locust movement in Kenya or the impact of natural disasters and a host of other applications of the data available from satellite images.

status.[utility].co.tt

Recently, there was thunder and lightning and heavy rainfall in Trinidad. Trees fell, and there was a range of property damage.

In cases like this, the aftermath can involve service outages across a number of service providers. For me, it was power. There was a bright spark, a loud bang and then silence.

Right after the first jumble of thoughts, I remembered you can call the power company to find out about outages. In this case, it was 800-BULB. The operator was prompt, he helpfully indicated that it was “a big problem that will take a long time to fix”.

When we awoke hours later, there was still no power, but added to that, I could not reach the power company on the phone. Either a lot of ringing or busy signals. And it made me think, calling to find out about the status of service is an approach we should retire.

From a cloud perspective, we can readily find out if a given provider is up or down by checking one of their status pages. Both Azure and GitHub comes to mind.

If we had access to the numbers, I expect the power company’s lines didn’t stop ringing until the day after the event. How did they handle the increased load and calls? That could have been lessened if there was an automated way their customers could get answers. Both at the phone level, and on the web.

I expect that some of the more advanced utility companies here in Trinidad already have internal systems that can readily provide this information, it’s therefore a matter of going the last mile and making it accessible to the public.

Of course, if utilities use a standard mechanism for reporting, then what can then happen is the status of a number of public utilities can be known to the nation, without having to make any calls.

What a day that would be.

Caribbean Developers’ Salary Survey – Redux

A developer survey I ran on Caribbean Developers came up recently, so I wanted to share what I learnt.

I saw on the Caribbean Developers’ Facebook group someone looking to run a salary survey:

I dig Matthew’s approach, because instead of doing the survey and trying to see if there’s interest, he’s looking to see if there’s any interest first and then leaning into the work.

From the looks of it, interest is low, maybe it will grow, but I wouldn’t be surprised if it doesn’t.

In 2018, I had a similar idea, for the same Caribbean Devs group. I may have been inspired by the StackOverflow Developer Survey and my own curiosity as it relates to the job market.

Here’s my survey results from back then:

Just about 40 submissions were made, which was instructive. I appreciated that if you do a salary survey in a small pool, it can be too revealing if you have access to each submission, even if the submissions are anonymized. If I were to do this again, I would bear that in mind, I might leave out a Company field, for example, in favor of Industry. I’d also not include a Job Title field and instead leave it at Category and perhaps Years in the industry. I might also include questions that get to the heart of project diversity, too.

So, this is mostly a cautionary tale about surveys like this. There are good points of guidance to get from economists and people in the social sciences to understand how to design things like this, beyond simply capturing the data.

Nurse Carter – a COVID19 hackathon winner

Nurse Carter in 2020.

So, when I built Nurse Carter years ago, I got the data from the Ministry of Health’s website in Trinidad and Tobago. I hadn’t updated the data since. Also, I built it with version 3 of the Microsoft Bot Framework.

Bot Framework updated a lot since then, and so did the health facility schedule. I took the opportunity to hit two birds with one code. I updated to the latest bot framework bits, changed from Node to dotnet and updated the health facility data.

To make it COVID-19 relevant, I included information about that as a menu option in Nurse Carter.

When I first did Nurse Carter, I feel like I had a lot more time – lol, I did because I was on “paternity leave” (it’s regular vacation I’m calling that). Now, in snatches of time, I found myself thinking “this is rell work!”

Nevertheless, I got to the submission line and Nurse Carter to a devpost hackathon and it was one of the top projects out of 1500 that were submitted! COVID-19 has brought a lot of heartache but I’m hoping that innovative approaches like Nurse Carter get more and more opportunities to deliver meaningful value in people’s lives.

That time when global quarantine liberated digital services. #CoronaEffect

There has been a sudden, dramatic step-wise improvement in digital services in Trinidad & Tobago.

You can place an order via WhatsApp for groceries in Trinidad & Tobago now. From one of the largest chains in the country.

Many government agencies are available in a real-time way on social media. Providing updates of a COVID and even non-COVID nature

A local bank has started to enable sending of money that the recipient has access to in 1 day, down from 4 to 5. The others, I expect are sure to follow.

I’m here for all of it. Maybe this activity is just merchants and other service providers responding to the change in environment. Everyone’s online, and the only way to get people to interact meaningfully with your service is by meeting them where they are at.

For now, I’ll list these services as I hear about them. I won’t celebrate too loudly, because, you know, many people have been clamoring for this for a long time.

Parse a pdf, quickly

  1. You have docker installed.
  2. You don’t have time to futz around with PDFs

If you have a PDF you’d like to get tables out of, tabula works a dream. When last I used it successfully, I was on my mac. I jumped on to a windows machine, and I just couldn’t get it to run.

*a few googles later*

I found a docker command that hosts a web API that gives access to tabula:

docker run -p 8080:8080 -e HOST=0.0.0.0 gavinkflam/tabula-api:1.0.0

At the readme for this container, the example was this:

curl -X POST -H 'Content-Type: multipart/form-data' \
  -F 'file=@my-pdf-file.pdf' -F 'guess=true' -F 'pages=all' \
  http://localhost:8080/api/extract

Thankfully, because of WSL, I just popped over into bash and I had tables in a flash 😎

I’m a student, where can I Azure? (freeness welcome)

Whenever I teach cloud technologies, I emphasize practical applications, even for the newest students. This really helps the topic come alive and allows the students to engage the material in a more realistic way.

As it turns out, a few of my friends reached out to me about Azure resources for learners, so, I thought I’d just jot down a few resources here.

Azure for students

So, a simple search online should point you here:

https://azure.microsoft.com/en-us/free/students/

One of the first questions that come up is “Am I worthy eligible?”, which is helpfully answered in the FAQ:

Azure for Students is available only to students who meet the following requirements. You must affirm that you are age 18 or older and attend an accredited, degree-granting two-year or four-year educational institution where you’re a full-time student in a STEM-related field. You must verify your academic status through your organization’s email address . This offer is not available for use in a massive open online course (MOOC) or in other professional trainings from for-profit organizations. The Azure for Students offer is limited to one per eligible student, is non-transferable, and cannot be combined with any other offers unless otherwise permitted by Microsoft. Azure for Students has a limited supply of activations and the offer is available only while supplies last. Each region has a limited number of Azure for Students offers available, and your geographical location may run out of supplies while the offer is still available in other regions. Microsoft may terminate this offer at our sole discretion. Other restrictions may apply.

Azure for faculty

Now, if you’re an educator, teaching assistant or other faculty, you can sign up for open source content – I’ve used this, it’s legit and the github repo is very active! As well as get a free Azure account for doing demos and showing how the stuff works.

Just plain old free Azure stuff

If you aren’t enrolled anywhere and just want to learn Azure stuff, there are also a lot of free-levels that you can use to get started.

https://azure.microsoft.com/en-us/free/

When you access this option, they’ll ask for a credit card number – this is for verification, they won’t charge you automatically unless you choose to upgrade. You have to choose to upgrade, it’s not going to sneak up on you.

Some azure services have levels that are always free, like Azure App Services, Azure Functions and Azure Kubernetes Service. There are others that are free for the first 12 months of usage. You can check out the full list here (scroll down).

So, there you go, get out there, get started, have fun!

Trinidad and Tobago Azure Users Group

We’re having the first of many (I hope) Microsoft Azure users meeting this Thursday (February 13).

Image result for microsoft azure
How do you Azure?

I’m excited by this because here in Trinidad, I’m grateful that in and around Teleios, we’ve been using Azure for years, and I’ve often wondered about who else around us is learning, playing and building on the platform. Nigel Sammy, Christian Anton and I will be sharing on Azure in general, Azure DevOps and Azure Arc. It’ll be a really nice spread of features, from the novel to the staid.

So, convince you’re boss, if you’re interested, we’ll be at the Microsoft Office on Mucurapo Road, from 9:00 to 11:00 am.

The event is free, so’s the breakfast and you can register here.

Three things about January

Just before I went back out to work in January, I had a fair sense of how I wanted to start the year. At work, I’d be diving in to Planner and refining some early 2020 goals that Anand and I had come up with. At projects, I’d be doing some arduino and bot stuff. All great.

But from January 2nd, I found myself very deep in legacy code from at least a decade ago. And I was working very late. It’s like time stood still. When my team and I raised our heads for a week, a change to the environment around that same code meant even more speelunking.

Out of this last few weeks I saw three things of note:

try
{
//actual code to do something meaningful
}
catch
{
//empty, meaningless catch block
}

  1. Stop using empty catch blocks.
    I haven’t had cause to leave dangly, useless catch blocks lying around for a few years, but I remembered when I used to. And boy was it a shocker. In the age of amazing tracing tools like App Insights, throwing away trace opportunities was like a slap in the face.
    When I showed a few developers at work some examples of empty catch blocks, they understood why it might have happened – I didn’t know what I wanted to do with the exception at the time – but my response was largely, “well, at least log it na”. At least. Even logging took a hit this month.
  2. Log with sense
    The worst kinds of exception message in a log is “An exception occurred”. For the most part, sometimes, when looking for the flow of a problem, I might just put a log statement to show that we got to a place in code. Sometimes, that place is an error handler. The way I see exceptions now, if you’re logging that it occurred, take some time to log what that block was trying to do, as well as what parameters are available. In a messaging system for example, simply a message Id is a big deal. So, first we log, then we log with sense.
  3. Sleep is very good.
    Not just sleep, but a firm commitment to not rush off with the first good idea you have when facing a problem. I have actually heard this stated another way, “Beware the danger of the first light”. While trying to debug our way out of the hole we were in, several ideas came up. Most recently, I had a great one that would have led to massive changes in the system. I was about to announce it, but one of the managers on our team cautioned me. She said it was Friday, sleep on it and see if you feel the same way on Monday.
    Monday came, I felt the same way. Until I spoke to one of the newest engineers on our team and realized we could achieve what I wanted to with a change to one module, as opposed to the whole system. That was good.
    Over the course of this month, there were a lot of challenges.

The TweetWhisperer

Today is GDG DevFest 2019 in Trinidad. The organizers put out a call for sessions, and I was happy to share one of the ideas that had been rolling around in my head for a while.

I Facebook in pirate, don’t @ me.

So, here’s the TL;DR: my idea was to take my likes on @Twitter and funnel them into Google Keep. Along the way, I’ll automatically categorize the tweets and then confirm that categorization via a chatbot. Simple stuff.

So simple, I didn’t even use Visio to diagram it.

What I actually did:

Twitter Likes

I made an Azure Function that would periodically poll my twitter account and get the last tweets that I liked. To do this, I had to create a developer account in twitter to get the appropriate creds. The function was pretty simple:

Categorizing Likes

In the DotNetConf keynote a few weeks ago, I saw an ML.NET demo and I got the idea to use it here, too.

ML.Net to build models (easy peasy)

All my notes

I pulled all my notes in keep to train an ML model. It was very easy, particularly because I used gkeepapi, an unsupported library for interacting with keep.

Doing this made me glad that I could think in terms of a bunch of cooperating functions, because the function to extract the notes from keep was written in python, while most everything else is in C#.

KeepIt: A function to get my notes from Google Keep

The funny thing is, I didn’t really need the model. Most of the things I stored in keep, were in one or two categories – not very good for modelling. I guess I hadn’t really realized that. To be honest, I was probably storing things in keep that I had a very high priority on, which turned out to be mostly cloud things. Go figure.

How the bot will help change things

So, I’m grabbing my tweets, categorizing them based on history and preference and I’m ready to store them, except, as I’ve shown above, my categorization is whack. Thus, I also made a chatbot, that will take my liked tweets and ask me to adjust the category I’ve labelled it as.

TweetWhisperer: Helping me categorize my tweets better.

So, with these three components, a likes-harvester, a categorizer and a chatbot, maybe, I’ll get better at returning to these topics that I have an interest in.