Jumping over hurdles to get to insights using Azure

On the way to getting some data to an Azure DB, I explored strategy in using entity framework that was such a cool timesaver, I thought I’d share it for #GlobalAzureBootcamp

I could have done this years ago. It’s just that sometimes a task feels so complex, daunting or mind-numbingly boring that you make do with alternatives until you just have to bite the bullet.

We work in SMPP at Teleios. It’s one of the ways you can connect to mobile carriers and exchange messages.  From time to time, we need to analyze the SMPP traffic we’re sending/receiving and typically, we use WireShark for this. That’s as easy as setting it up to listen on a port and then filtering for the protocol we care about. Instead of actively monitoring with WireShark, we may at times use dumpcap to get chunks of files with network traffic for analysis later on.

What we found was that analyzing file by file was tedious, we wanted to analyze a collection of files at once. We’d known of cloud-based capture analysis solutions for a while, but they tended to focus on TCP and maybe HTTP. Our solution needed to be SMPP-based. So, we decided to build a rudimentary SMPP-based database to help with packet analysis.

That’s where the mind-numbingly boring work came in. In the SMPP 3.4 spec, a given SMPP PDU can contain 150 fields. The need for analysis en masse was so great that we had to construct that database. But this is where the ease of using modern tools jumped in.

I got a list of the SMPP fields from the WireShark site.  In the past, I would have then gone about crafting the SQL that represented those fields as columns in a table. But now, in the age of ORM, I made a class in C#. And if you’re following along from the top, I created a project in Visual Studio, turned on Entity Framework migrations and added a DataContext. From there, it was plain sailing, as I just needed to introduce my new class to my DataContext and push that migration to my db.

It probably took me 30 minutes to go from the list of 150 fields on WireShark to being able to  create the database without the necessary tables. Now, where does Azure come into all of this?

Each capture file we collect could contain thousands of messages. So, in my local development environment, when I first tried to ingest all those to my database, my progress bar told me I’d be waiting a few days.  With Azure, I rapidly provisioned a database and had it’s structure in place with this gem:

Database.Migrate();

That is, from the Entity Framework DataContext class, I called the Database.Migrate() method and any newly set up db would have my latest SMPP table. From there, I provisioned my first 400 DTU Azure SQL database and saw my progress bar for ingestion drop from days to hours.

With a database that size, queries over thousands of rows went by reasonably fast and gave us confidence that we’re on the right path.

We’re also working on a pipeline that automates ingestion of new capture files, very similar to what I did last year my Azure Container Instances project.

So, for #GlobalAzureBootcamp 2019, I’m glad that we were able to step over the hurdle of tedium into richer insights in our message processing.

Making a map.

I like how WordPress’ default on these blog posts is “tell your story”. LOL.

Recently, I described wanting to use Azure App Insights for dependency tracking. I started by looking at documentation on it. I ended up using this, along with this video.

As I started looking into the first place to drop in dependency notification, which was where I create my Azure Container Instance. The tutorial’s code suggested dropping this fragment into a finally around where the dependency was, like so:

From: https://docs.microsoft.com/en-us/azure/azure-monitor/app/api-custom-events-metrics#trackdependency

Then I found myself wondering if for the dependencies I was going to track, wouldn’t a C# attribute be really good. Then I fell down a rabbit hole of when to create attributes.

And then I emerged out of that hole, attribute-less.

So, after I applied guidance from the Application Insights document, I colored my external operations with track dependency calls and ended up with this image:

Azure Application Insights application map for my solution

This does what I wanted, which is to visualize the components and dependencies of my solution, with information on success/failure and duration of the calls.

I’d like to do more with this.

Conquering complexity with a map

Last year, I worked with a researcher to develop a really cool, complex Azure solution to augment her work flow with some data. She needed to ingest a large volume of data, clean it up, run some AI on that and then present it. Typical data science activities that she wanted to run in the cloud.

I implemented the flow using several components including Azure Container Instances, Event Grid, Azure ML, Cosmos DB and Azure Functions. My daily drive at work doesn’t necessarily let me play in all those spaces at once, so I felt really glad to see all of those pieces work together.

Deploying took a bit more work as I wanted to make that as straightforward as possible. Thus, I used the Azure Fluent SDK that I was fanboying about across a few posts in 2018.

After everything was deployed though, I found visibility into the system was a bit of a stretch. My researcher colleague wanted to easily know where things were at in a given iteration of the process. I didn’t have a lot of solutions for that, so it was mostly email alerts.

That is, until I learnt about Azure Application map from two of my colleagues at Teleios – Ikechi, in Ops and Anand in Engineering.

It’s a part of Application Insights and lets you track dependencies between various services in an Azure solution. So, just out of the box, you can view the success of calls between web sites and web services and databases and other components. Going further, you can even add other components and dependencies to your app. That got me thinking. Maybe I can use Azure Application Map to display the various components of the solution and track issues in a visual, at-a-glance way?

I’m going to check it out.

Project VILMA

Sometimes, I get home and just wish I could say to my virtual machine in the cloud, “magico presto, turn on!” and it’s on and ready for me to remote into and do things. I wanted to do things to make that happen, but time and procrastination happens to many. Thankfully, there was an upcoming developer gathering that I used as the catalyst to actually build a system that would work, almost like magic.

So, last Sunday, the Port of Spain chapter of GDG (Google Developer Groups) held a developer event, #GDGPOS #DevFest. They reached out to the local developer community for interesting projects and I responded with a proposal to build something that would work in the way I described.

gdg-presenters
GDGPOS Presenters

My proposal got accepted and I spent a few weeks building out the idea. My whole solution involved using my Google mini to turn my virtual machine on or off.

To do that, I created a Google Action on the Google Actions console. I had played around with Actions before, but this would be different. I have been making most of my conversational agents using Microsoft’s Bot FrameworkBot Framework, so a lot of the concepts were familiar to me, from things like Intents, to Utterances and even the use of webhooks. For this action, I largely had to focus on just one Intent – the one that would hear a command for a VM state change and execute. Overall, the system would look like this:

VILMA-diagram

  • Creating the action

So, I created this custom Intent that took me to Dialogflow, Google’s interactive tool for building conversational interfaces. There, I created a custom intent, ChangeVMState.

ChangeVMState would receive messages and figure out if to turn a VM on or off. The messages could be in a range of formats like:

  • turn on/off
  • power on/off
  • shutdown/start up the vm

They would resolve to the ChangeVMState intent. All messages sent to ChangeVMState was then forwarded to my webhook. I deployed the webhook as a function in Azure.

The code to execute the functions is pretty straightforward. One function receives the request and queues it on an Azure Storage Queue.  Azure functions provides a really simple infrastructure for doing just that. 

I mean, this is the whole method: 

The item being put on the queue – the desired VM state – is just a variable being set. 

Another function in Azure will then take up the values in the queue and will start or stop the VM based on state. Again, a pretty simple bit of code. 

I’m using the Azure Fluent Management SDK to start/stop a VM

So, finally, after the VM is put into the desired state, an email is sent either saying the VM is off or that it’s on and an RDP file is included. Ideally, I wanted to have the Google Assistant I was using notify me when the VM got up and running, but I just couldn’t get push notifications working – which is why I ended up with email. 

Thus,  I ended up with a Google Action, that I eventually called VILMA Agent (at first, I was calling it Shelly-Ann).  I could say to my Google Mini, “OK, Google, tell VILMA Agent, turn on” and I’d get an email with an RDP file.

The code for the functions part of VILMA is up here on GitHub. 

“Pacers”

In my running group I declared I was going to do this year’s UWI Half Marathon in 11:00 minute miles. My friend roundly berated me for aiming so slow, us having run together and him just knowing I can do a bit better.

Nothing’s wrong with that time if that’s where your training and skill have taken you. But he was like, you do the thing at 9:00 last year and reach 11:00 this year?! LOL, he was right. But I had a bag of excuses, less time to train, distractions, tiredness.

The man said don’t come and talk to me if you run this thing at no 11.

I had a fairly safe race planned. Because truly, my training wasn’t where I wanted it this year (also, lol, this is usually the case). But I planned to play some nice chill music, run at 10:30 for the first half and then try and maintain an even 10 on the way back down.

I started off too quickly.

By the time I got to my first half-mile, with my chill music playing, Endomondo told me I was at 8:something a mile. 🙂 Not good – based on my safe outlook on the race. A little after the mile, I realized I was not going to play this according to plan at all.

This is just after mile 5. Me and the pacer became friends this day.

I ran with the pacer. The 9:00-mile pacer. I saw him coming up with his jolly, flag self just after the first mile, I did a quick self-diagnostic check and determined I felt good at that pace and just went with it.

Pacers feel like the best thing innovation to the UWI half in a long time! They aren’t there to win, not there to get a medal, just there to be a beacon, and not even a silent one. Dude was hailing out people he knew, encouraging his growing following and just being literally a living flag of hope.

So, I ran with the pacer all the way to the half-mark. He had to turn off for a bit and I did not sustain my 9:00. I ran too fast, I ran too slow, but I had enough gas and my new music kicked in at the right time. I listened to both recent hip hop albums from Congress Music Factory that seem to be perfect for running as well as spiritual sustenance.

But all that delight started to wane at mile 10. It became a familiar slog as the sun came up, my muscles’ tiredness became more vocal and essentially, I started to lose focus. My pacer buddy came back, like a good Pied Piper with his merry crew. But this time, I couldn’t answer his call.

By mile 11, I did get to that 11:00-mile pace, with a single goal: keep running. Or jogging. But not walking. I knew from experience that the walk would feel REALLY GOOD, but do a lot of damage to my momentum. So I kept running and by mile 12 got the surge I needed to finish the race.

I ended up having a really decent average pace of 9:17-mile. Not a bad result.

WhatsApp Image 2018-10-28 at 7.45.16 PM

PS:

At the end of the race in the runner’s tent, I was waiting for a massage. Next to me sat this “elderly” dude. He told me he’s been running for longer than I’ve been alive. His race time? 1:37:00.

 

Danger Zones

TL; DR

I’ve built a map of the location updates from the Ministry of Works and Transport of Trinidad and Tobago based on flooding and where was/is impassable. You van view it here.

“Technical” details

That tweet above is kind of how I got the idea in my head to build out an example of the approach.

When I sat down to do create a version of a good approach, I had all kinds of options in my mind. Should it be rendered on the client or server side? React or Angular? Should I use Google Maps, Leaflet & MapBox or something else? How would I generate the data?  Should I try and parse some tweets? What’s the fastest way to get data? Who has the data?

Since I didn’t want to spend all evening in analysis paralysis, I just dove in and began pulling things together. I had recently set up a new dev environment, so my regular tools for some ideas weren’t restored yet. No node, npm or React was set up. So I started downloading packages, installers and tools.

And then I remembered glitch! I literally paused mid environment setup and jumped onto searching in glitch. Glitch is like online development environment that comes prepackaged with the resources you need to get up and running with with minimal fuss. Now, you have to have a sense of what you want to build and what tech to use. Which I did. A few searches later, I found a great starting point, something that already had the Leaflet stuff built in.

Having the base I wanted, I needed to get the content of these tweets represented as geojson:

Again, numerous options, parsers to write and just ideas swirling around. But while spelunking online for stuff to use, I found geojson.io – a WYSIWIG for generating geojson. I had to handcode the stuff, switching between Google Maps, Open Streetmaps and Waze but I just wanted an early result.

And I got it: a map that presents the information that @mowtgovtt tweeted about the state of impassable regions in the country.

 

Funky Azure Functions

Let’s talk about watering plants.

When I was younger, in my family, I was assigned the task of watering the flowering plants around the house. Thinking back on it now, there was easily 50 plants of all shapes and sizes. So, I would have to shuffle around the yard, bucket in hand, dipping and watering. Some plants would get two dips, others one. I couldn’t use the hose, because that might damage the roots of the younger plants. I hated it.

Ever the creative, I used to come up with outlandish ideas to solve the predicament. Sadly, I never implemented any of them. Thus, I was left to water these plants by hand.

Last week, for Caribbean Developer Week, I came up with a demo, featuring Azure Functions, that is the nearest to a solution to my plant watering needs back then that I have ever come.

I built three Azure Functions:

  1. Setup Waterer
  2. GuidEnqueuer
  3. Plant Waterer

Setup Waterer actually created more Azure Functions. Those would be Timer functions, each potentially able to run their own schedule.

GuidEnqueuer, alas poorly named, but good at pretending to be a plant food source, would receive an Http post and enqueue it. Plant Waterer would pick this up and display on a console. No actual plants benefited from this demo.

As I gushed previously, I created the Setup Waterer function on top of the Azure Fluent SDK and it worked fine. Functions making functions. That’s what I wanted to show really, and things worked well.

The code is available on my repo here.