Back to the Sky: Processing Satellite Data Using Cloud Computing

From time to time, I work with researchers on projects outside of the day to day, forms-over-databases stuff. Mind you, I like a nice form over a cool data repository as much as the next guy, but it’s definitely cool to stretch your arms and do something more.

So, when Dr. Ogundipe approached me about cloudifying her satellite data processing, I had to do a lot of research. She had a processing pipeline that featured ingesting satellite images, doing some data cleanup, then analyzing those results using machine learning. Everything ran on local machines in her lab, and she knew she would run into scaling problems.

Early on, I decided to use a containerized approach to some of the scripting that was performed. The python scripts were meant to run in Windows, but I had an easier go at the time getting Linux containers up and running, so I went with that. Once the container was in good order, I stored the image in the Azure Container Registry and then fired it up using an Azure Container Instance.

Like a good story, I had started in the middle – with the data processing. I didn’t know how I would actually get non-test data into the container. Eventually, I settled on using Azure Files. Dr. Ogundipe would upload the satellite images via a network drive mapped to storage in the cloud. Since I got to have some fun with the fluent SDK in Azure a while back, I used it to build an orchestrator of sorts.

Once the orchestrator had run, it would have fed the satellite images into the container. Output from the container was used to run models stored in Azure ML. Instead of detailing all the steps, this helpful diagram explains the process well:

Super simple.

No, not that diagram.

The various cloud resources used to process satellite data in Azure.

So, I shared some of this process at a seminar Dr. Ogundipe held to talk about the work she does, and how her company, Global Geo-Intelligence Solutions Ltd uses a pipeline like this to detect locust movement in Kenya or the impact of natural disasters and a host of other applications of the data available from satellite images.

I’m a student, where can I Azure? (freeness welcome)

Whenever I teach cloud technologies, I emphasize practical applications, even for the newest students. This really helps the topic come alive and allows the students to engage the material in a more realistic way.

As it turns out, a few of my friends reached out to me about Azure resources for learners, so, I thought I’d just jot down a few resources here.

Azure for students

So, a simple search online should point you here:

https://azure.microsoft.com/en-us/free/students/

One of the first questions that come up is “Am I worthy eligible?”, which is helpfully answered in the FAQ:

Azure for Students is available only to students who meet the following requirements. You must affirm that you are age 18 or older and attend an accredited, degree-granting two-year or four-year educational institution where you’re a full-time student in a STEM-related field. You must verify your academic status through your organization’s email address . This offer is not available for use in a massive open online course (MOOC) or in other professional trainings from for-profit organizations. The Azure for Students offer is limited to one per eligible student, is non-transferable, and cannot be combined with any other offers unless otherwise permitted by Microsoft. Azure for Students has a limited supply of activations and the offer is available only while supplies last. Each region has a limited number of Azure for Students offers available, and your geographical location may run out of supplies while the offer is still available in other regions. Microsoft may terminate this offer at our sole discretion. Other restrictions may apply.

Azure for faculty

Now, if you’re an educator, teaching assistant or other faculty, you can sign up for open source content – I’ve used this, it’s legit and the github repo is very active! As well as get a free Azure account for doing demos and showing how the stuff works.

Just plain old free Azure stuff

If you aren’t enrolled anywhere and just want to learn Azure stuff, there are also a lot of free-levels that you can use to get started.

https://azure.microsoft.com/en-us/free/

When you access this option, they’ll ask for a credit card number – this is for verification, they won’t charge you automatically unless you choose to upgrade. You have to choose to upgrade, it’s not going to sneak up on you.

Some azure services have levels that are always free, like Azure App Services, Azure Functions and Azure Kubernetes Service. There are others that are free for the first 12 months of usage. You can check out the full list here (scroll down).

So, there you go, get out there, get started, have fun!

Trinidad and Tobago Azure Users Group

We’re having the first of many (I hope) Microsoft Azure users meeting this Thursday (February 13).

Image result for microsoft azure
How do you Azure?

I’m excited by this because here in Trinidad, I’m grateful that in and around Teleios, we’ve been using Azure for years, and I’ve often wondered about who else around us is learning, playing and building on the platform. Nigel Sammy, Christian Anton and I will be sharing on Azure in general, Azure DevOps and Azure Arc. It’ll be a really nice spread of features, from the novel to the staid.

So, convince you’re boss, if you’re interested, we’ll be at the Microsoft Office on Mucurapo Road, from 9:00 to 11:00 am.

The event is free, so’s the breakfast and you can register here.

Making a map.

I like how WordPress’ default on these blog posts is “tell your story”. LOL.

Recently, I described wanting to use Azure App Insights for dependency tracking. I started by looking at documentation on it. I ended up using this, along with this video.

As I started looking into the first place to drop in dependency notification, which was where I create my Azure Container Instance. The tutorial’s code suggested dropping this fragment into a finally around where the dependency was, like so:

From: https://docs.microsoft.com/en-us/azure/azure-monitor/app/api-custom-events-metrics#trackdependency

Then I found myself wondering if for the dependencies I was going to track, wouldn’t a C# attribute be really good. Then I fell down a rabbit hole of when to create attributes.

And then I emerged out of that hole, attribute-less.

So, after I applied guidance from the Application Insights document, I colored my external operations with track dependency calls and ended up with this image:

Azure Application Insights application map for my solution

This does what I wanted, which is to visualize the components and dependencies of my solution, with information on success/failure and duration of the calls.

I’d like to do more with this.

Conquering complexity with a map

Last year, I worked with a researcher to develop a really cool, complex Azure solution to augment her work flow with some data. She needed to ingest a large volume of data, clean it up, run some AI on that and then present it. Typical data science activities that she wanted to run in the cloud.

I implemented the flow using several components including Azure Container Instances, Event Grid, Azure ML, Cosmos DB and Azure Functions. My daily drive at work doesn’t necessarily let me play in all those spaces at once, so I felt really glad to see all of those pieces work together.

Deploying took a bit more work as I wanted to make that as straightforward as possible. Thus, I used the Azure Fluent SDK that I was fanboying about across a few posts in 2018.

After everything was deployed though, I found visibility into the system was a bit of a stretch. My researcher colleague wanted to easily know where things were at in a given iteration of the process. I didn’t have a lot of solutions for that, so it was mostly email alerts.

That is, until I learnt about Azure Application map from two of my colleagues at Teleios – Ikechi, in Ops and Anand in Engineering.

It’s a part of Application Insights and lets you track dependencies between various services in an Azure solution. So, just out of the box, you can view the success of calls between web sites and web services and databases and other components. Going further, you can even add other components and dependencies to your app. That got me thinking. Maybe I can use Azure Application Map to display the various components of the solution and track issues in a visual, at-a-glance way?

I’m going to check it out.

Project VILMA

Sometimes, I get home and just wish I could say to my virtual machine in the cloud, “magico presto, turn on!” and it’s on and ready for me to remote into and do things. I wanted to do things to make that happen, but time and procrastination happens to many. Thankfully, there was an upcoming developer gathering that I used as the catalyst to actually build a system that would work, almost like magic.

So, last Sunday, the Port of Spain chapter of GDG (Google Developer Groups) held a developer event, #GDGPOS #DevFest. They reached out to the local developer community for interesting projects and I responded with a proposal to build something that would work in the way I described.

gdg-presenters
GDGPOS Presenters

My proposal got accepted and I spent a few weeks building out the idea. My whole solution involved using my Google mini to turn my virtual machine on or off.

To do that, I created a Google Action on the Google Actions console. I had played around with Actions before, but this would be different. I have been making most of my conversational agents using Microsoft’s Bot FrameworkBot Framework, so a lot of the concepts were familiar to me, from things like Intents, to Utterances and even the use of webhooks. For this action, I largely had to focus on just one Intent – the one that would hear a command for a VM state change and execute. Overall, the system would look like this:

VILMA-diagram

  • Creating the action

So, I created this custom Intent that took me to Dialogflow, Google’s interactive tool for building conversational interfaces. There, I created a custom intent, ChangeVMState.

ChangeVMState would receive messages and figure out if to turn a VM on or off. The messages could be in a range of formats like:

  • turn on/off
  • power on/off
  • shutdown/start up the vm

They would resolve to the ChangeVMState intent. All messages sent to ChangeVMState was then forwarded to my webhook. I deployed the webhook as a function in Azure.

The code to execute the functions is pretty straightforward. One function receives the request and queues it on an Azure Storage Queue.  Azure functions provides a really simple infrastructure for doing just that. 

I mean, this is the whole method: 

[FunctionName("GoogleActions")]
public static IActionResult Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
[Queue("changevmstate")] out string state,
ILogger log)
{
state = null;
try
{
string requestBody = new StreamReader(req.Body).ReadToEnd();
var data = JsonConvert.DeserializeObject<Request>(requestBody);
log.LogInformation(requestBody);
var query= data.queryResult.queryText;
dynamic actionResult = new ExpandoObject();
string result = string.Empty;
if (data.queryResult.intent.displayName == "ChangeVMState")
{
state = data.queryResult.parameters.vmstate;
result = $"{state} is going to be applied, Irwin";
actionResult.fulfillmentText = result;
}
actionResult.source = botId;
var output = JsonConvert.SerializeObject(actionResult,
Formatting.Indented, new AtAtJsonConverter(typeof(ExpandoObject)));
return output != null
? (ActionResult)new JsonResult(actionResult)
: new BadRequestObjectResult("ERROR: "+output);
}
catch (Exception ex)
{
log.LogError(ex, ex.Message);
return new BadRequestObjectResult("Badness");
}
}
view raw Function.cs hosted with ❤ by GitHub
The item being put on the queue – the desired VM state – is just a variable being set. 

Another function in Azure will then take up the values in the queue and will start or stop the VM based on state. Again, a pretty simple bit of code. 

public Tuple<string,string> SetState(string state)
{
var vmName = VirtualMachineName;
var client = AzureClientId;
var key = AzureClientKey;
var tenant = AzureTenantId;
var creds = new AzureCredentialsFactory()
.FromServicePrincipal(client, key, tenant, AzureEnvironment.AzureGlobalCloud);
var subscriptionId = AzureSubscriptionId;
var azure = Microsoft.Azure.Management.Fluent.Azure.Authenticate(creds).WithSubscription(subscriptionId);
var allVMs = azure.VirtualMachines.List();
var vmCurrent = (from vm in allVMs where vm.Name == vmName select vm).First();
switch (state)
{
case "on":
vmCurrent.Start();
break;
case "off":
vmCurrent.PowerOff();
break;
default:
break;
}
var message = new Tuple<string, string>(vmCurrent
.GetPrimaryNetworkInterface().PrimaryIPConfiguration.GetPublicIPAddress().IPAddress
,vmCurrent.PowerState.Value);
return message;
}
view raw VMManagement.cs hosted with ❤ by GitHub
I’m using the Azure Fluent Management SDK to start/stop a VM

So, finally, after the VM is put into the desired state, an email is sent either saying the VM is off or that it’s on and an RDP file is included. Ideally, I wanted to have the Google Assistant I was using notify me when the VM got up and running, but I just couldn’t get push notifications working – which is why I ended up with email. 

Thus,  I ended up with a Google Action, that I eventually called VILMA Agent (at first, I was calling it Shelly-Ann).  I could say to my Google Mini, “OK, Google, tell VILMA Agent, turn on” and I’d get an email with an RDP file.

The code for the functions part of VILMA is up here on GitHub

Provisioning some test storage accounts for class

I wanted to create a few storage accounts for students in my class to complete an assignment featuring Event Sourcing and Material Views.

So, here’s what I did.

Download/install the latest azure command line interface (cli).
(While doing this, I realized I could have just used the cloud shell. I soldiered on with the dl)

Create a resource group to contain the accounts we’d need.

#Prior to doing this, ensure that user is logged in
# 'az login' works
#Then, if you have multiple subscriptions attached to account, select the appropriate one using:
# 'az account set –subscription <name or id>'
#command below:
az group create –name COMP6905A2Storage #name I used

Create the accounts and output the storage account keys
The command to make a single storage account is pretty straightforward:

#ensure logged in to azure
#ensure default subscription is desired one
az storage account create –name comp69052017a2test \ #test storage account
–resource-group COMP6905A2Storage \#test resource group
–location eastus –sku Standard_LRS \
–encryption blob

But I wanted to also output the keys and display them on a single line. The command to get the keys after the account is created is this:

az storage account keys list –account-name comp69052017a2test –resource-group COMP6905A2Storage

So, I used the jq program in bash to parse the json result and display both keys on a line. Thus, I created a script that would create the accounts and then output their storage account keys.
This is the script that produced the accounts and keys:

for number in {1..20}
do
account=comp69052017a2
account+=$number
az storage account create –name $account –resource-group COMP6905A2Storage –location eastus –sku Standard_LRS –encryption blob | jq ".name"
az storage account keys list –account-name $account –resource-group COMP6905A2Storage | jq '.[].value' | tr -s '\n' ','
done

Overall, the longest part of the exercise was dealing with the way the files were being saved in windows vs how they were being saved and read by bash. But the accounts were created and class can get on with assignment 2.

Teaching Cloud Technologies (2016/2017)

We just concluded another run of the Cloud Technologies course at the University of the West Indies. This course is part of the MSc Computer Science program.

As lecturer, I had to come up with the course outline as well as the content. In so doing, I get the opportunity to refresh what we talk about as well as how we approach assignments.

This course comprised of ten modules:

  1. Intro to Cloud Technologies
  2. Cloud computing infrastructure
  3. Virtualization
  4. Big data
  5. Cloud development patterns
  6. Cloud resource management
  7. IaaS Automation
  8. Microservices
  9. IoT
  10. Cloud for Research

Though our primary cloud platform is Microsoft Azure, students are free during assignments and project submission to use other cloud providers.

One of the assignments involves virtual machine scale sets and containers. The draught goes like this:

Z. Zanko Systems provide sales processing systems for large commercial banks.

They receive more than 5 million JSON requests per hour (revised to 300,000).
Each request must be stored in permanent storage. The format of the request is:
“{“TransactionID”:”1″,”UserId”:”A1″,”SellerID”:”S1″,”Product Name”:”Financial Trap”,”Sale Price”:1000000,”Transaction Date”:” “}”

You have been hired as a System Developer by Zanko. You have access to VMs whose capacity is equal to that of A1 VMs in Azure IaaS or Containers of similar capacity.

Develop a mechanism to generate the requests your system faces.

Design and implement a solution using a container-based approach or a
virtual machine-based one to process 5 million requests in an hour.
For your receivers introduce a failure rate.
Store the occurrence of failures.
Justify how you chose to store and monitor failures.

Though most students can build this out using azure, one enterprising student chose to use AWS and reading his submission was a nice view of getting this done using Amazon’s resources vs Azure.

This year, Microsoft put a halt to the Azure for Education Academic grant, but did have a number of other ways for students to get into cloud, including DreamSpark and other offers.

The project component this year changed a bit, too. In the two years prior, we asked students to build working cloud services themselves. This year, we asked them to propose a cloud service that featured understanding of:

  • Cloud service definition
  • Cloud service models
  • Cloud delivery models
  • Cloud for research
  • Cloud development with a regional focus

We saw some excellent solutions that we hope to hear more about in the future.