GitHub Copilot: A young Jarvis?

I recently watched yet another time travel TV series on Netflix. This time out of South Korea. I forgot the name, so in looking it up I realized that time-travel K-dramas are very much a thing, lol. Anyway, it’s called, “Sisyphus: The Myth”. It comes to mind because the writers of the show were fond of having characters repeat the quote,

The future is already here – it’s just not evenly distributed

William Gibson

I mean, for a time travel show, it’s fine. And I think they probably got what they were going after since here I am, quoting them, quoting Gibson. But look, I wrote about Jarvis about two years ago. Sort of wondering when will that kind of tech be available. Of having a sort of programmatic au pair who would come alongside and help not just with syntax and formatting, which admittedly, has existed for a long time, but with coming up with the actual code based on your intent.

In that blog post, I didn’t put a time horizon on it, but I thought, “maybe soonish”. This week though, I realized it’s already here.

I just got access to the still-in-beta GitHub Copilot.

It’s been blowing my mind and I didn’t even read everything on the tin, I just dove right in. In following the first example, it involved creating a method to get the difference between two dates:

First example … writing a method with GitHub Copilot’s help.

Again, I didn’t read everything, so I wasn’t sure what would happen when I began the method definition, but things fell into place quickly. I typed the “function calculateBetweenDates(” and sure enough, just like I have grown accustomed to with IntelliSense, I saw a prompt.

But, unlike IntelliSense, these prompts weren’t based on any methods I had written or that was available in some framework/library that had been loaded. These prompts came from Copilot’s body of knowledge.

Now, excited, I wanted to use an example of something I would do from time to time.

For a lot of my current projects there’s a “sign in to your cloud provider” step. So, that’s the method I went with. I wrote, “function signInToAzure(” and after a tab, Copilot has inserted the full method.

Playing just a bit more, in the generated method, I noticed the need for something related to the AAD_AUTH_URL. So, I started a method, “function GetAzureADAuthUrl” and voila, Copilot understood and gave me the method.

These are simple examples, but I really dig how it works.

I’ve got an AFRAME project I wanted to take this on a spin with, so I’ll see it in action in more detail, but this is very groovy already.

Now, just like Tony in Iron Man 2, the question of if he should even be the hero, wielding the suit while being the child and prime beneficiary of a massively successful arms company, the moral questions are in no short supply with Copilot. This includes the licenses of the code used to train Copilot, and whether they permit it to exist in its current form.

Nat Friedman, GitHub CEO weighed in on some of those issues here, essentially concluding,

We expect that IP and AI will be an interesting policy discussion around the world in the coming years, and we’re eager to participate!

Nat Friedman, HackerNews

I’m still in experiment-and-play mode with Copilot, observing where the technology goes, but I’m mindful of the arguments about its existence and will keep tracking that.

Here’s to Jarvis, he’s a teenager now, but already showing promise.

#MSBuild 2021 Table Talk: How it went

In a recent post I mentioned moderating a table talk at this year’s MSBuild along with Erik Kleefeldt.

By design, it’s meant to be like a hallway chat with a bunch of people about tech we’ve been using/playing with/excited about. This hallway was huge. 318 people turned up on Teams to talk about extending it in some way or other.

Erik and I met with Anna, Andy and Joey from Microsoft in the preceding weeks, to get oriented, nail down a flow and just get a backstage peak at some of coordination it takes to pull off an event like Build. (Hint: It’s a lot!).

We had a good idea about what we would do, I’d introduce, he’d conclude and stuff like that. And then when the meeting started, I had jitters, my machine had jitters and I wondered if we would actually get started at all. But then, I told all my new friends that I was super nervous and everything just settled down.

Top things we spoke about:

• Fluid Framework
• Viva Connections
• Upcoming Meetings and Calling API

As a hallway chat, like in real life, there’s no recording, but those topics are great areas to explore further. I’m definitely looking forward to the new media API – to get me out of being stuck in the middle of my current project.

Overall, this was a lot of fun, Build was plenty vibes & plenty action and I’ve got a lot of unpacking to do from my backpack.

In media res

I’ve been playing with an idea in Microsoft Teams for a few months now. It hasn’t yet borne fruit, but I’ve decided its time to move on, and maybe revisit it in a few more months with fresh eyes.

This sort of breaks with how I’ve considered writing posts on here. I like each post related to a creative activity to represent a singular finished work. I can refine things and come back, but each should say, “I did this and here’s the result”. But I’m writing about being in the middle of things, largely as an exercise in discipline, but also, as a clear record for myself of what I’ve done.

So what was it?

Scott Hanselman had this post on Twitter about making credits appear at or near the end of a Teams meeting. From the time he put it up, I was pretty clear on how he might have done it.

My Teams meetings always end SO awkwardly and it's not clear when to leave.

So I made live end credits.

(no post credits scene) pic.twitter.com/2pBThNBi2x

— Scott Hanselman (@shanselman) February 3, 2021

Genesis of my idea for a Microsoft Teams End Credits bot came from this.

I think apart from the fact that we all become video presenters last year, we also became much more familiar with OBS. And the approach he used could be done with OBS. Which he described here.

In fact, I think I saw an earlier tweet from him about a dude doing essentially a transparent board overlay with OBS, too. OBS to me, feels easy to use, but when you put everything together, it could feel like a bit of work. You have to set up your scenes, fill them with the right layers, and hang everything together, just so.

So, not hard, but somewhat involved. Since I’d been experimenting with the audio and video streams of Teams calls, I could see how a similar thing could possibly be done in Teams directly. Which would let me yell, “Look ma! No OBS!”, while achieving the same functionality.

Quite a few of these experiments begin with messing around with a sample, here and there. Call it SDD – Sample Driven Development. I picked up where the HueBot Teams sample left off. It’s one that let’s you create a bot that grabs a speaker’s video stream in a call and overlay it with a given hue – red, green or blue. I’d gotten that to work. And last time I played with that sample, I was able to send music down into a Teams meeting using a set of requests to a chatbot.

Now, I wanted to essentially overlay on a given video stream, the same credits info that I saw from Scott’s OBS trick.

I am currently still in that rabbit hole.

Yes, I was able to access the video stream based on the sample. I even got to the point of overlaying text. But pushing that back down to the call for everyone? Various shades of failure.

• 

• 

• 

• 

• 

The first image is essentially straight out of the sample, where guidance was provided on how to extract a bitmap image from the video stream, which is otherwise formatted in NV12. The other images in the carousel are what appeared in Teams, with various degrees of resizing but always having a blue hue.

	/// <summary>
	/// Transform NV12 to bmp image so we can view how is it looks like. Note it's not NV12 to RBG conversion.
	/// </summary>
	/// <param name="data">NV12 sample data.</param>
	/// <param name="width">Image width.</param>
	/// <param name="height">Image height.</param>
	/// <param name="logger">Log instance.</param>
	/// <returns>The <see cref="Bitmap"/>.</returns>
	public static Bitmap TransformNv12ToBmpFaster(byte[] data, int width, int height, IGraphLogger logger)
	{
	Stopwatch watch = new Stopwatch();
	watch.Start();
	var bmp = new Bitmap(width, height, PixelFormat.Format32bppPArgb);
	var bmpData = bmp.LockBits(
	new Rectangle(0, 0, bmp.Width, bmp.Height),
	ImageLockMode.ReadWrite,
	PixelFormat.Format32bppRgb);
	var uvStart = width * height;
	for (var y = 0; y < height; y++)
	{
	var pos = y * width;
	var posInBmp = y * bmpData.Stride;
	for (var x = 0; x < width; x++)
	{
	var vIndex = uvStart + ((y >> 1) * width) + (x & ~1);
	//// https://msdn.microsoft.com/en-us/library/windows/desktop/dd206750(v=vs.85).aspx
	//// https://en.wikipedia.org/wiki/YUV
	var c = data[pos] – 16;
	var d = data[vIndex] – 128;
	var e = data[vIndex + 1] – 128;
	c = c < 0 ? 0 : c;
	var r = ((298 * c) + (409 * e) + 128) >> 8;
	var g = ((298 * c) – (100 * d) – (208 * e) + 128) >> 8;
	var b = ((298 * c) + (516 * d) + 128) >> 8;
	r = r.Clamp(0, 255);
	g = g.Clamp(0, 255);
	b = b.Clamp(0, 255);
	Marshal.WriteInt32(bmpData.Scan0, posInBmp + (x << 2), (b << 0) | (g << 8) | (r << 16) | (0xFF << 24));
	pos++;
	}
	}
	bmp.UnlockBits(bmpData);
	watch.Stop();
	logger.Info($"Took {watch.ElapsedMilliseconds} ms to lock and unlock");
	return bmp;
	}

view raw TransformNv12ToBmpFaster.cs hosted with ❤ by GitHub

This code essentially does the transformation.

I’m currently stuck with that blue hue. 😕.

So, since the sample only had a one-way transformation of NV12 to bitmap, not having any experience with that, I speelunked around the web for a solution. Normally that would mean some drive-by [StackOverflow]ing for a whole method, but that got me as far as those blue hues.

Literally, the method I got from S/O let me convert BMP to some kind of NV12, but not something that Teams quite liked.

	private byte [] getYV12(int inputWidth, int inputHeight, Bitmap scaled) {
	int [] argb = new int[inputWidth * inputHeight];
	scaled.getPixels(argb, 0, inputWidth, 0, 0, inputWidth, inputHeight);
	byte [] yuv = new byte[inputWidth*inputHeight*3/2];
	encodeYV12(yuv, argb, inputWidth, inputHeight);
	scaled.recycle();
	return yuv;
	}
	private void encodeYV12(byte[] yuv420sp, int[] argb, int width, int height) {
	final int frameSize = width * height;
	int yIndex = 0;
	int uIndex = frameSize;
	int vIndex = frameSize + (frameSize / 4);
	int a, R, G, B, Y, U, V;
	int index = 0;
	for (int j = 0; j < height; j++) {
	for (int i = 0; i < width; i++) {
	a = (argb[index] & 0xff000000) >> 24; // a is not used obviously
	R = (argb[index] & 0xff0000) >> 16;
	G = (argb[index] & 0xff00) >> 8;
	B = (argb[index] & 0xff) >> 0;
	// well known RGB to YUV algorithm
	Y = ( ( 66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
	U = ( ( –38 * R – 74 * G + 112 * B + 128) >> 8) + 128;
	V = ( ( 112 * R – 94 * G – 18 * B + 128) >> 8) + 128;
	// YV12 has a plane of Y and two chroma plans (U, V) planes each sampled by a factor of 2
	// meaning for every 4 Y pixels there are 1 V and 1 U. Note the sampling is every other
	// pixel AND every other scanline.
	yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
	if (j % 2 == 0 && index % 2 == 0) {
	yuv420sp[uIndex++] = (byte)((V<0) ? 0 : ((V > 255) ? 255 : V));
	yuv420sp[vIndex++] = (byte)((U<0) ? 0 : ((U > 255) ? 255 : U));
	}
	index ++;
	}
	}
	}

view raw yv12.java hosted with ❤ by GitHub

I converted this Java method to c#.

Part of the conversion meant reading up on YUV. The Java method focused on YV12. Teams needed the stream to be NV12. Their differences are summarized here:

NV12

Related to I420, NV12 has one luma “luminance” plane Y and one plane with U and V values interleaved.

In NV12, chroma planes (blue and red) are subsampled in both the horizontal and vertical dimensions by a factor of 2.

For a 2×2 group of pixels, you have 4 Y samples and 1 U and 1 V sample.

It can be helpful to think of NV12 as I420 with the U and V planes interleaved.

Here is a graphical representation of NV12. Each letter represents one bit:

For 1 NV12 pixel: YYYYYYYY UVUV

For a 2-pixel NV12 frame: YYYYYYYYYYYYYYYY UVUVUVUV

For a 50-pixel NV12 frame: Y×8×50 (UV)×2×50

For a n-pixel NV12 frame: Y×8×n (UV)×2×n

FROM: VideoLan on YUV#NV12

	public void BMPtoNV12(byte[] yuv420sp, byte[] argb, int width, int height)
	{
	int frameSize = width * height;
	int yIndex = 0;
	int uvIndex = frameSize;
	uint a;
	int R, G, B, Y, U, V;
	int index = 0;
	for (int j = 0; j < height; j++)
	{
	//int index = width * j;
	for (int i = 0; i < width; i++)
	{
	a = (argb[index] & 0xff000000) >> 24; // a is not used obviously
	R = (argb[index] & 0xff0000) >> 16;
	G = (argb[index] & 0xff00) >> 8;
	B = (argb[index] & 0xff) >> 0;
	// well known RGB to YUV algorithm
	Y = (( 66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
	U = ((–38 * R – 74 * G + 112 * B + 128) >> 8) + 128;
	V = ((112 * R – 94 * G – 18 * B + 128) >> 8) + 128;
	//NV12
	//Related to I420, NV12 has one luma "luminance" plane Y and one plane with U and V values interleaved.
	//In NV12, chroma planes(blue and red) are subsampled in both the horizontal and vertical dimensions by a factor of 2.
	//For a 2×2 group of pixels, you have 4 Y samples and 1 U and 1 V sample.
	//It can be helpful to think of NV12 as I420 with the U and V planes interleaved.
	//Here is a graphical representation of NV12.Each letter represents one bit:
	//For 1 NV12 pixel: YYYYYYYY UVUV
	//For a 2 – pixel NV12 frame: YYYYYYYYYYYYYYYY UVUVUVUV
	//For a 50 – pixel NV12 frame: Y×8×50(UV)×2×50
	//For a n – pixel NV12 frame: Y×8×n(UV)×2×n
	yuv420sp[yIndex++] = (byte)((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
	if (j % 2 == 0 && i % 2 == 0)
	{
	yuv420sp[uvIndex++] = (byte)((U < 0) ? 0 : ((U > 255) ? 255 : U));
	yuv420sp[uvIndex++] = (byte)((V < 0) ? 0 : ((V > 255) ? 255 : V));
	}
	index++;
	}
	}
	}

view raw BMPtoNV12.cs hosted with ❤ by GitHub

Converted the YV12 approach to NV12.

Even though I modified a method’s to produce NV12 from a BMP array, no joy. And this after much tinkering..

Eventually, I even tried using the OpenCV project, but that just led to green splotches all over.

Thus, I’m stuck. I still love the idea, but I’ve poured way too many hours into the experiment at this stage. I’m looking forward to Microsoft’s Build this week. Maybe I’ll find some helpful soul to set me on the straight and narrow.

One Marvelous Scene – Tony and Jarvis

My video editing skills are what you might call, “Hello World” level. So, I’m not interested in bringing down the average as it were, in terms of the quality content over on the official “One Marvelous Scene” playlist on YouTube.

I dig what those storytellers did a lot. But as far as I can tell, even though there are 67 videos on that list, they missed a key one. It was from when Tony had returned home in Iron Man 1 and was building out the new suit, working with Jarvis.

Back when IM1 came out, I remember being delighted by the movie, just for the flow of it, just as a young developer then, seeing a portrayal of a “good programmer” was refreshing. And he was getting things wrong. Making measuring mistakes or pushing things out too quickly. He had to return to the drawing board a lot. It was messy. Just like writing code.

And while doing this, he was working along with Jarvis. An AI helper with “personality”. One that he could talk to in a normal way and move his work along.

In 2008, that was still viewed as fanciful, in terms of a developer experience, but even then I was somewhat seriously considering its potential, which is why I even asked about voice commands for VS on StackOverflow 👇🏾

Jump forward to 2019 and bot framework is a thing, the StackOverflow bot is also a thing and Team Visual Studio introduced IntelliCode as this year’s build conference.

So, as a scene in a movie, Tony talking with Jarvis helps us understand Tony’s own creative process, but for me, it gave glimpses into the near future for how I could be building applications. And that’s marvelous.