Category Archives: Programming

Related to some aspect of programming, software development, related tools, or supporting technologies, related standards, etc.

Stupid Azure Trick #6 – A CORS Toggler Command-line Tool for Windows Azure Blobs

1 Reply

[Edit: I originally accidentally published an old draft. The draft went out to all email subscribers and was public for around 90 minutes. Fixed now.]

In the most recent Stupid Azure Trick installment, I explained how one could host a 1000 visitor-per-day web site for one penny per month. Since then I also explained my choice to use CORS in that same application. Here I will dig into specifically using CORS with Windows Azure.

I also show how the curl command line tool can be helpful to examine CORS properties in HTTP headers for a blob service.

I also will briefly describe a simple tool I built that could quickly turn CORS on or off for a specified Blob service – the CORS Toggler. The CORS Toggler (in its current simple form) was useful to me because of two constraints that were true for my scenario:

  • I was only reading files from the Windows Azure Blob Service. When just reading, pre-flight request doesn’t matter when you are just reading. Simplification #1.
  • I didn’t care whether the blob resource is publicly available, rather than just available to my application. So the CORS policy was to open to any caller (‘*’). Simplification #2.

These two simplifications mean that the toggler knew what it meant to enable CORS (open up for reading to all comers) and to disable. (Though it is worth noting that opening up CORS to any caller is probably a common scenario. Also worth noting that tool could easily extended to support a whitelist for allowed domains or other features.)

First, here’s the code for the toggler – there are three files here:

  1. Driver program (Console app in C#) – handles command line params and such and then calls into the …
  2. Code to perform simple CORS manipulation (C# class)
  3. The above two and driven (in my fast toggler) through the third file (command line batch file) which passes in the storage keys and storage account name for the service I was working with
@echo off
D:\dev\path\to\ToggleCors.exe azuremap 123abcYourStorageKeyIsHere987zyx== %1
echo.
echo FOR COMPARISON QUERY THE nocors SERVICE (which never has CORS set)
echo.
D:\dev\path\to\ToggleCors.exe nocors 123abcYourStorageKeyIsHere987zyx== -q
echo.
PowerShell -Command Get-Date
view raw cors-toggler.bat hosted with ❤ by GitHub
using System;
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage.Blob;
using Microsoft.WindowsAzure.Storage.Shared.Protocol;
using Newtonsoft.Json;
namespace BlobCorsToggle
{
public class SimpleAzureBlobCorsSetter
{
public CloudBlobClient CloudBlobClient { get; private set; }
public SimpleAzureBlobCorsSetter(Uri blobServiceUri, StorageCredentials storageCredentials)
{
this.CloudBlobClient = new CloudBlobClient(blobServiceUri, storageCredentials);
}
public SimpleAzureBlobCorsSetter(CloudBlobClient blobClient)
{
this.CloudBlobClient = blobClient;
}
/// <summary>
/// Set Blob Service CORS settings for specified Windows Azure Storage Account.
/// Either sets to a hard-coded set of values (see below) or clears of all CORS settings.
///
/// Does not check for any existing CORS settings, but clobbers with the CORS settings
/// to allow HTTP GET access from any origin. Non-CORS settings are left intact.
///
/// Most useful for scenarios where a file is published in Blob Storage for read-access
/// by any client.
///
/// Can also be useful in conjunction with Valet Key Pattern-style limited access, as
/// might be useful with a mobile application.
/// </summary>
/// <param name="clear">if true, clears all CORS setting, else allows GET from any origin</param>
public void SetBlobCorsGetAnyOrigin(bool clear)
{
// http://msdn.microsoft.com/en-us/library/windowsazure/dn535601.aspx
var corsGetAnyOriginRule = new CorsRule();
corsGetAnyOriginRule.AllowedOrigins.Add("*"); // allow access to any client
corsGetAnyOriginRule.AllowedMethods = CorsHttpMethods.Get; // only CORS-enable http GET
corsGetAnyOriginRule.ExposedHeaders.Add("*"); // let client see any header we've configured
corsGetAnyOriginRule.AllowedHeaders.Add("*"); // let clients request any header they can think of
corsGetAnyOriginRule.MaxAgeInSeconds = (int)TimeSpan.FromHours(10).TotalSeconds; // clients are safe to cache CORS config for up to this long
var blobServiceProperties = this.CloudBlobClient.GetServiceProperties();
if (clear)
{
blobServiceProperties.Cors.CorsRules.Clear();
}
else
{
blobServiceProperties.Cors.CorsRules.Clear(); // replace current property set
blobServiceProperties.Cors.CorsRules.Add(corsGetAnyOriginRule);
}
this.CloudBlobClient.SetServiceProperties(blobServiceProperties);
}
public void DumpCurrentProperties()
{
var blobServiceProperties = this.CloudBlobClient.GetServiceProperties();
var blobPropertiesStringified = StringifyProperties(blobServiceProperties);
Console.WriteLine("Current Properties:\n{0}", blobPropertiesStringified);
}
internal string StringifyProperties(ServiceProperties serviceProperties)
{
// JsonConvert.SerializeObject(serviceProperties) for whole object graph or
// JsonConvert.SerializeObject(serviceProperties.Cors) for just CORS
return Newtonsoft.Json.JsonConvert.SerializeObject(serviceProperties, Formatting.Indented);
}
}
}
view raw SimpleAzureBlobCorsSetter.cs hosted with ❤ by GitHub
using System;
using System.Diagnostics;
using System.IO;
using Microsoft.WindowsAzure.Storage.Auth;
namespace BlobCorsToggle
{
class Program
{
static string clearFlag = "-clear";
static string queryFlag = "-q";
static void Main(string[] args)
{
if (args.Length == 0 || !ValidCommandArguments(args))
{
#region Show Correct Usage
if (args.Length != 0 && !ValidCommandArguments(args))
{
var saveForegroundColor = Console.ForegroundColor;
Console.ForegroundColor = ConsoleColor.Red;
Console.Write("\nINVALID COMMAND LINE OR UNKNOWN FLAGS: ");
foreach (var arg in args) Console.Write("{0} ", arg);
Console.WriteLine();
Console.ForegroundColor = saveForegroundColor;
}
Console.WriteLine("usage:\n{0} <storage acct name> <storage acct key> [{1}]",
Path.GetFileNameWithoutExtension(Environment.GetCommandLineArgs()[0]),
clearFlag);
Console.WriteLine();
Console.WriteLine("example setting:\n{0} {1} {2}",
Path.GetFileNameWithoutExtension(Environment.GetCommandLineArgs()[0]),
"mystorageacct",
"lala+aou812SomERAndOmStrINglOlLolFroMmYStORaG3akounT314159265358979323ilIkEpi==");
Console.WriteLine();
Console.WriteLine("example clearing:\n{0} {1} {2} {3}",
Path.GetFileNameWithoutExtension(Environment.GetCommandLineArgs()[0]),
"mystorageacct",
"lala+aou812SomERAndOmStrINglOlLolFroMmYStORaG3akounT314159265358979323ilIkEpi==",
clearFlag);
if (Debugger.IsAttached)
{
Console.WriteLine("\nPress any key to exit.");
Console.ReadKey();
}
#endregion
}
else
{
var storageAccountName = args[0];
var storageKey = args[1];
var blobServiceUri = new Uri(String.Format("https://{0}.blob.core.windows.net", storageAccountName));
var storageCredentials = new StorageCredentials(storageAccountName, storageKey);
var blobConfig = new SimpleAzureBlobCorsSetter(blobServiceUri, storageCredentials);
bool query = (args.Length == 3 && args[2] == queryFlag);
if (query)
{
blobConfig.DumpCurrentProperties();
return;
}
blobConfig.DumpCurrentProperties();
Console.WriteLine();
bool clear = (args.Length == 3 && args[2] == clearFlag);
blobConfig.SetBlobCorsGetAnyOrigin(clear);
Console.WriteLine("CORS Blob Properties for Storage Account {0} have been {1}.",
storageAccountName, clear ? "cleared" : "set");
Console.WriteLine();
blobConfig.DumpCurrentProperties();
}
}
private static bool ValidCommandArguments(string[] args)
{
if (args.Length == 2) return true;
if (args.Length == 3 && (args[2] == clearFlag || args[2] == queryFlag)) return true;
return false;
}
}
}
view raw ToggleCors.cs hosted with ❤ by GitHub

One simple point to highlight – CORS properties are simply available on the Blob service object (and would be same for Table or Queue service within Storage):

image

Yes, this is a very simple API.

Showing the Service Object Contents

For those interested in the contents of these objects, here are a few ways to show content of properties (in code) before turning on CORS and after. (The object views are created using the technique I described my post on using JSON.NET as an object dumper that’s Good Enough™.)

DUMPING OBJECT BEFORE CORS ENABLED (just CORS properties):

{“Logging”:{“Version”:”1.0″,”LoggingOperations”:0,”RetentionDays”:null},”Metrics”:{“Version”:”1.0″,”MetricsLevel”:0,”RetentionDays”:null},”HourMetrics”:{“Version”:”1.0″,”MetricsLevel”:0,”RetentionDays”:null},”Cors”:{“CorsRules”:[]},”MinuteMetrics”:{“Version”:”1.0″,”MetricsLevel”:0,”RetentionDays”:null},”DefaultServiceVersion”:null}

DUMPING OBJECT AFTER CORS ENABLED:

{“Logging”:{“Version”:”1.0″,”LoggingOperations”:0,”RetentionDays”:null},”Metrics”:{“Version”:”1.0″,”MetricsLevel”:0,”RetentionDays”:null},”HourMetrics”:{“Version”:”1.0″,”MetricsLevel”:0,”RetentionDays”:null},”Cors”:{“CorsRules”:[{“AllowedOrigins”:[“*”],”ExposedHeaders”:[“*”],”AllowedHeaders”:[“*”],”AllowedMethods”:1,”MaxAgeInSeconds”:36000}]},”MinuteMetrics”:{“Version”:”1.0″,”MetricsLevel”:0,”RetentionDays”:null},”DefaultServiceVersion”:null}

image

DUMPING OBJECT BEFORE CORS ENABLED (but including ALL properties):

Current Properties:
{“Logging”:{“Version”:”1.0″,”LoggingOperations”:0,”RetentionDays”:null},”Metrics
“:{“Version”:”1.0″,”MetricsLevel”:0,”RetentionDays”:null},”HourMetrics”:{“Versio
n”:”1.0″,”MetricsLevel”:0,”RetentionDays”:null},”Cors”:{“CorsRules”:[]},”MinuteM
etrics”:{“Version”:”1.0″,”MetricsLevel”:0,”RetentionDays”:null},”DefaultServiceV
ersion”:null}

DUMPING OBJECT AFTER CORS ENABLED (but including ALL properties):

Current Properties:
{“Logging”:{“Version”:”1.0″,”LoggingOperations”:0,”RetentionDays”:null},”Metrics
“:{“Version”:”1.0″,”MetricsLevel”:0,”RetentionDays”:null},”HourMetrics”:{“Versio
n”:”1.0″,”MetricsLevel”:0,”RetentionDays”:null},”Cors”:{“CorsRules”:[{“AllowedOr
igins”:[“*”],”ExposedHeaders”:[“*”],”AllowedHeaders”:[“*”],”AllowedMethods”:1,”M
axAgeInSeconds”:36000}]},”MinuteMetrics”:{“Version”:”1.0″,”MetricsLevel”:0,”Rete
ntionDays”:null},”DefaultServiceVersion”:null}

image

Using ‘curl’ To Examine CORS Data:

curl -H "Origin: http://example.com&quot; -H "Access-Control-Request-Method: GET" -H "Access-Control-Request-Headers: X-Requested-With" -X OPTIONS --verbose http://azuremap.blob.core.windows.net/maps/azuremap.geojson
view raw check-cors hosted with ❤ by GitHub

image

CURL OUTPUT BEFORE CORS ENABLED:

D:\dev\github>curl -H “Origin: http://example.com” -H “Access-Control-Request-Method: GET” -H “Access-Control-Request-Headers: X-Requested-With” -X OPTIONS –verbose http://azuremap.blob.core.windows.net/maps/azuremap.geojson

* Adding handle: conn: 0x805fa8
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* – Conn 0 (0x805fa8) send_pipe: 1, recv_pipe: 0
* About to connect() to azuremap.blob.core.windows.net port 80 (#0)
*   Trying 168.62.32.206…
* Connected to azuremap.blob.core.windows.net (168.62.32.206) port 80 (#0)
> OPTIONS /maps/azuremap.geojson HTTP/1.1
> User-Agent: curl/7.31.0
> Host: azuremap.blob.core.windows.net
> Accept: */*
> Origin: http://example.com
> Access-Control-Request-Method: GET
> Access-Control-Request-Headers: X-Requested-With
>
< HTTP/1.1 403 CORS not enabled or no matching rule found for this request.
< Content-Length: 316
< Content-Type: application/xml
* Server Blob Service Version 1.0 Microsoft-HTTPAPI/2.0 is not blacklisted
< Server: Blob Service Version 1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: 04402242-d4a7-4d0c-bedc-ff553a1bc982
< Date: Sun, 26 Jan 2014 15:08:11 GMT
<
<?xml version=”1.0″ encoding=”utf-8″?><Error><Code>CorsPreflightFailure</Code><Message>CORS not enabled or no matching rule found for this request.
RequestId:04402242-d4a7-4d0c-bedc-ff553a1bc982
Time:2014-01-26T15:08:12.0193649Z</Message><MessageDetails>No CORS rules matches this request</MessageDetails></Error>*
Connection #0 to host azuremap.blob.core.windows.net left intact

CURL OUTPUT AFTER CORS ENABLED:

D:\dev\github>curl -H “Origin: http://example.com” -H “Access-Control-Request-Method: GET” -H “Access-Control-Request-Headers: X-Requested-With” -X OPTIONS –verbose http://azuremap.blob.core.windows.net/maps/azuremap.geojson
* Adding handle: conn: 0x1f55fa8
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* – Conn 0 (0x1f55fa8) send_pipe: 1, recv_pipe: 0
* About to connect() to azuremap.blob.core.windows.net port 80 (#0)
*   Trying 168.62.32.206…
* Connected to azuremap.blob.core.windows.net (168.62.32.206) port 80 (#0)
> OPTIONS /maps/azuremap.geojson HTTP/1.1
> User-Agent: curl/7.31.0
> Host: azuremap.blob.core.windows.net
> Accept: */*
> Origin: http://example.com
> Access-Control-Request-Method: GET
> Access-Control-Request-Headers: X-Requested-With
>
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
* Server Blob Service Version 1.0 Microsoft-HTTPAPI/2.0 is not blacklisted
< Server: Blob Service Version 1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: d4df8953-f8ae-441b-89fe-b69232579aa4
< Access-Control-Allow-Origin: http://example.com
< Access-Control-Allow-Methods: GET
< Access-Control-Allow-Headers: X-Requested-With
< Access-Control-Max-Age: 36000
< Access-Control-Allow-Credentials: true
< Date: Sun, 26 Jan 2014 16:02:25 GMT
<
* Connection #0 to host azuremap.blob.core.windows.net left intact

Resources

A new version of the Windows Azure Storage Emulator (v2.2.1) is now in Preview. This release has support for “2013-08-15” version of Storage which includes CORS (and JSON and other) support.

Overall description of Azure Storage’s CORS Support:

http://msdn.microsoft.com/en-us/library/windowsazure/dn535601.aspx

REST API doc (usually the canonical doc for any feature, though in code it is easily accessed with the Windows Azure SDK for .NET)

http://msdn.microsoft.com/en-us/library/windowsazure/hh452235.aspx

A couple of excellent posts from the community on CORS support in Windows Azure Storage:

[This is part of a series of posts on #StupidAzureTricks, explained here.]

Choosing CORS over JSONP over Inline… and Lessons Learned Using CORS

3 Replies

I recently created a very simple client-only (no server-side code) that loads needed data dynamically. In order to access data from another storage location (in my case the data came from a Windows Azure Blob), the application needed to make a choice: how to load the data.

It really came down to three choices:

  1. Load the data synchronously as the page loaded using an Inline script tag
  2. Load the data asynchronously as part of initial page load using JSONP
  3. Load the data asynchronously as part of initial page load using CORS

All 3 options effectively work within the Same Origin Policy (SOP) sandbox security measures that browsers implement. If access is not coming from a browser (but from, say, curl or a server application), SOP has no effect. SOP is there to protect end users from web sites that might not behave themselves.

Option 1 would be to basically have a hardcoded script tag load the data. One disadvantage of this is put perfectly by Douglas Crockford: “A <script src="url"></script> will block the downloading of other page components until the script has been fetched, compiled, and executed.” This means that the page will block while the data is loaded, potentially making the initial load appear a bit more visually chaotic. Also, if this technique is the only mechanism for loading data, once the page is loaded, the data is never refreshed, a potentially severe limitation for some applications; in the very old days, the best we could do was periodically trigger a full-page refresh, but that’s not state-of-the-art in 2014.

Option 2 would be to load the data asynchronously using JSONP. This is a fine solution from a user experience point of view: the page structure is first loaded, then populated once the data arrives. The client invokes the request using the XMLHttpRequest object in JavaScript.

Option 3 would be to load the data asynchronously using CORS. This offers essentially the identical user experience as option 2 and also relies on the XMLHttpRequest object.

Options 1 and 2 require that the data be encapsulated in JavaScript code. For option 2 with JSONP the convention is a function (often named callback) that simply returns a JSON object. The client making the call will then need to execute the function to get at the data. Option 1 has slightly more flexibility and could be simply a data structure declared with a known name like var mapData = ... which the client can access directly.

Option 3 with CORS is able to return the data directly. In that regard it is a little tiny bit more efficient since no bubble-wrap is needed – and is a lot safer since you are not executing a JavaScript function returned returned by a potentially untrusted server.

JSONP is not based on any official standard, but is common practice. CORS is a standard that is supported in modern browsers and comes with granular access policies. As an example, CORS policies can be set to allow access from a whitelist of domains (such as paying customers), while disallowing from any other domain.

For all three options there needs to be coordination between the client and the server since they need to agree on how the data is packaged for transmission. For CORS, this also requires browser support (see chart below). All options require that JavaScript is enabled in the client browser.

Summarizing CORS, JSONP, Inline

The following summary compares key qualities.

Inline JavaScript JSONP CORS Comments
Synchronous or Async Synchronous Async Async
Granular domain-level security no no yes In any of the three, you could also implement an authorization scheme. This is above and beyond that.
Risk no yes no JSONP requires that you execute a JavaScript function to get at the data. Neither of the other two approaches require that. There’s an extra degree of caution needed for JSONP data sources outside of your control.
Efficiency on the wire close close most efficient Both Inline and JSONP both wrap your data in JavaScript constructs. These add a small amount of overhead. Depending on what you are doing, these could add up. But minor.
Browser support full full partial
Server support full full partial Servers need to support the CORS handshake with browsers to (a) deny disallowed domains, and (b) to give browsers the information they need to honor restrictions
Supported by a Standard no no yes
Is it the future no no yes Safer. Granular security. Standardized. Max efficiency.

Lessons Learned Using CORS

Yes, my simple one-page map app (described here) ended up using CORS. In large part since it is mature, and the browser support (see below) was sufficient.

Reloading Browser Pages: In debugging, CTRL-F5 is your friend in Chrome, Firefox, and IE if you want to clear the cache and reload the page you are on. I did this a lot as I was continually enabled and disabling CORS on the server to test out the effects.

Overriding CORS Logic in CHROME: It turns out that Chrome normally will honor all CORS settings. This is what most users will see. Let’s call this “civilian mode” for Chrome. But there’s also a developer mode – which you enable by running chrome with the chrome.exe –disable-web-security parameter. It was initially confused since it seemed Chrome’s CORS support didn’t work, but of course it did. This is one of the perils of living with a software nerd; my wife had used my computer and changed this a long time ago when she needed to build some CORS features, and I never knew until I ran into the perplexing issue.

Handling CORS Rejection: Your browser may not not let your JavaScript code know directly that a remote call was rejected due to a CORS policy. Some browsers silently map 404 to 0 if against a CORS-protected resource. You’ll see this mentioned in the code for httpGetString.js (if you look at my sample code).

Testing CORS from curl: Helped by a post on StackOverflow, I found it very handy to look at CORS headers from the command line. Note that you need to provide SOME origin in the request for it to be valid CORS, but here’s the command that worked for my cloud-host resource (you should also be able to run this same command):

curl -H “Origin: http://localhost” -H “Access-Control-Request-Method: GET” -H “Access-Control-Request-Headers: X-Requested-With” -X OPTIONS –verbose http://azuremap.blob.core.windows.net/maps/azuremap.geojson

Browser Support for CORS

To understand where CORS support stands with web browsers, this fantastic site http://caniuse.com/cors offers a nice visual showing CORS support across today. A corresponding chart for JSONP is not needed since it works within long-standing capabilities.

image

Resources

http://en.wikipedia.org/wiki/Same-origin_policy

My simple one-page map app is described here. That page includes a link to a running instance and its source code is easily viewed with View Source.

http://blog.auth0.com/2014/01/27/ten-things-you-should-know-about-tokens-and-cookies/#preflight

Stupid Azure Trick #5 – Got a Penny? Run a Simple Web Site 100% on Blob Storage for a Month – Cost Analysis Provided

9 Replies

Suppose you have a simple static web site you want to publish, but your budget is small. You could do this with Windows Azure Storage as a set of blobs. The “simple static” qualifier rules out ASP.NET and PHP and Node.js – and anything that does server-side processing before serving up a page. But that still leaves a lot of scenarios – and does not preclude the site from being interactive or loading external data using AJAX and behaving like it is dynamic. This one does.

Check out the web site at http://azuremap.blob.core.windows.net/apps/bingmap-geojson-display.html.

image

You may recognize the map from an earlier post that showed how one could visualize Windows Azure Data Center Regions on a map. It should look familiar because this web site uses the exact same underlying GeoJSON data used earlier, except this time the map implementation is completely different. This version has JavaScript code that loads and parses the raw GeoJSON data and renders it dynamically by populating a Bing Maps viewer control (which is also in JavaScript).

But the neat part is there’s only JavaScript behind the scenes. All of the site’s assets are loaded directly from Windows Azure Blob Storage (plus Bing Maps control from an external location).

Here’s the simple breakdown. There is the main HTML page (the URL specifies that directly), and that in turn loads the following four JavaScript files:

  1. http://ecn.dev.virtualearth.net/mapcontrol/mapcontrol.ashx?v=7.0 – version 7.0 of the Bing Map control
  2. httpGetString.js – general purposes data fetcher (used to pull in the GeoJSON data)
  3. geojson-parse.js – application-specific to parse the GeoJSON data
  4. bingmap-geojson-display.js – application-specific logic to put elements from the GeoJSON file onto the Bing Map

I have not tried this to prove the point, but I think that to render on, say, Google Maps, the only JavaScript that would need to change would be bingmap-geojson-display.js (presumably replaced by googlemap-geojson-display.js).

Notice that the GeoJSON data lives in a different Blob Storage Container here:  http://azuremap.blob.core.windows.net/maps/azuremap.geojson. We’ll get into the details in another post, but in order for this to work – in order for …/apps/bingmap-geojson.html to directly load a JSON data file from …/maps/azuremap.geojson – we enabled CORS for the Blob Service within the host Windows Azure Storage account.

Costs Analysis

Hosting a very low-cost (and low-complexity) web site as a few blobs is really handy. It is very scalable and robust. Blob Storage costs come from three sources:

  1. cost of data at rest – for this scenario, probably Blob Blobs and Locally Redundant Storage would be appropriate, and the cost there is $0.068 per GB / month (details)
  2. storage transactions – $0.005 per 100,000 transactions (details – same as above link, but look lower on the page) – where a storage transaction is (loosely speaking) a file read or write operation
  3. outbound data transfers (data leaving the data center) – first 5 GB / month is free, then there’s a per GB cost (details)

The azuremap web site shown earlier weighs in at under 18 KB and is spread across 5 files (1 .html, 3 .js, 1 .geojson). If we assume a healthy 1000 hits a day on our site, here’s the math.

  • We have around 1000 x 31 = 31,000 visits per month.
  • Cost of data at rest would be 18 KB x $0.068 / GB = effectively $0. Since storage starts at less than 7 cents per GB and our data is 5 orders of magnitude smaller, the cost is too small to meaningfully measure.
  • Storage transactions would be 31,000 x 5 (one per file in our case) x $0.005 / 100,000 = $0.00775, or a little more than 3/4 of a penny in US currency per month, around 9 cents per year, or $1 every 11 years.
  • Outbound data transfer total would be 31,000 x 18 KB = 560 MB, which is around 1/10th of the amount allowed for free, so there’d be no charge for that.

So our monthly bill would be for less than 1 penny (less than US$0.01).

This is also a good (though very simple) example of the sort of cost analysis you will need to do when understanding what it takes to create cloud applications or migrate from on-premises to the cloud. The Windows Azure Calculator and information on lower-cost commitment plans may also prove handy.

Alternative Approaches

Of course in this day and age, for a low-cost simple site it is hard to beat Windows Azure Web Sites. There’s an entirely free tier there (details) – allowing you to save yourself nearly a penny every month. That’s pretty good since Benjamin Franklin, one of America’s founding fathers, famously quipped A penny saved is a penny earned!.BenFranklinDuplessis.jpg

Windows Azure Web Sites also has other features – your site can be in PHP or ASP.NET or Node.js or Python. And you can get continuous deployment from GitHub or Bitbucket or TFS or Dropbox or others. And you get monitoring and other features from the portal. And more.

But at least you know you can host in blob storage if you like.

[This is part of a series of posts on #StupidAzureTricks, explained here.]

Dumping objects one property at a time? A Pretty-Printer for C# objects that’s Good Enough™

3 Replies

Over the years, I’ve written a lot of code that simple dumps out an object’s properties. Sometimes this is for debugging, sometimes it is for output to Console.WriteLine. But a lot of those cases are plain old BORING, and the only reason I end up typing in obj.foo, obj.bar, and obj.gizmo is that I was too lazy to figure out how to easily stringify an entire object at a time – so I kept doing it one property (and sub-property (and sub-sub-property ..)) at a time.

I know that ToString() is supposed to help out (in .NET at least), but you probably noticed how uncommon it is for this to be usefully implemented.

There’s a better way.

A Pretty-Printer for C# objects that’s usually Good Enough™

The simple way to dump objects that’s often good enough (but not always good enough) is to use Json.NET’s object serializer.

Add Json.NET using NuGet, then using a code snippet like the following to dump out an object named someObject :

Console.WriteLine(Newtonsoft.Json.JsonConvert.SerializeObject(
someObject, Formatting.Indented));

That’s pretty much it. That’s the whole trick.

Note: You can use Formatting.None instead of Formatting.Indented if you want a more compact output (though harder to read).

Here are a couple of reasons why is isn’t always better:

  • You get the WHOLE object graph (no filtering – but see this and this)
  • Fields appear in JSON in the order they appear in the object – you don’t get to change it
  • Not easily massaged (e.g., do you want only a certain number of decimal places?)
  • (Probably more since I just started using this…)

Useful in other languages

This hack applies to any language that supports JSON serializers and formatters. For example, in Python, check out the json module.

Examples in C#

Here is are a couple of examples using a CORS tool I was fiddling with. In these examples, the serviceProperties object is of type ServiceProperties, a class from the Windows Azure Storage SDK for .NET.

Dump Just CORS:
Newtonsoft.Json.JsonConvert.SerializeObject(
serviceProperties.Cors, Formatting.Indented);

"Cors": {
   "CorsRules": [
   {
      "AllowedOrigins": [
      "*"
   ],
   "ExposedHeaders": [
      "*"
   ],
   "AllowedHeaders": [
      "*"
   ],
   "AllowedMethods": 1,
   "MaxAgeInSeconds": 36000
   }
   ]
}

Dump Entire Properties object:

Newtonsoft.Json.JsonConvert.SerializeObject(serviceProperties, Formatting.Indented);

{
   "Logging": {
      "Version": "1.0",
      "LoggingOperations": 0,
      "RetentionDays": null
   },
   "Metrics": {
      "Version": "1.0",
      "MetricsLevel": 0,
      "RetentionDays": null
   },
      "HourMetrics": {
      "Version": "1.0",
      "MetricsLevel": 0,
      "RetentionDays": null
   },
   "Cors": {
      "CorsRules": [
      {
         "AllowedOrigins": [
            "*"
         ],
         "ExposedHeaders": [
            "*"
         ],
         "AllowedHeaders": [
            "*"
         ],
         "AllowedMethods": 1,
         "MaxAgeInSeconds": 36000
      }
      ]
   },
   "MinuteMetrics": {
      "Version": "1.0",
      "MetricsLevel": 0,
      "RetentionDays": null
   },
   "DefaultServiceVersion": null
}

To use another concrete example, consider a simple program that I wrote a while back called DumpAllWindowsCerts.cs. The program just iterates through the Certificate Store on the current machine and dumps out a bunch of information. It uses Console.WriteLine statements to do this.

To compare the old and new outputs, I jumped to the LAST Console.WriteLine statement in the file and changed it to a JsonConvert.SerializeObject statement. Here’s what happened.

Note that the old Console.WriteLine statement was very limited since the contents of these objects varied a lot, so I had kept is simple (I didn’t know what I wanted, really). But the JSON output is pretty reasonable.

————————————————– Console.WriteLine

OID = Key Usage
OID = Basic Constraints [Critical]
OID = Subject Key Identifier
OID = CRL Distribution Points
...

————————————————– JSON.NET

{
   "KeyUsages": 198,
   "Critical": false,
   "Oid": {
      "Value": "2.5.29.15",
      "FriendlyName": "Key Usage"
   },
   "RawData": "AwIBxg=="
}
{
   "CertificateAuthority": true,
   "HasPathLengthConstraint": false,
   "PathLengthConstraint": 0,
   "Critical": true,
   "Oid": {
      "Value": "2.5.29.19",
      "FriendlyName": "Basic Constraints"
   },
   "RawData": "MAMBAf8="
}
{
   "SubjectKeyIdentifier": "DAED6474149C143CABDD99A9BD5B284D8B3CC9D8",
   "Critical": false,
   "Oid": {
      "Value": "2.5.29.14",
      "FriendlyName": "Subject Key Identifier"
   },
   "RawData": "BBTa7WR0FJwUPKvdmam9WyhNizzJ2A=="
}
{
   "Critical": false,
   "Oid": {
      "Value": "2.5.29.31",
      "FriendlyName": "CRL Distribution Points"
   },
   "RawData": "MDkw...9iamVjdC5jcmw="
}

…

Talk: Windows Azure Web Sites are PaaS 2.0

4 Replies

Last night I had the chance to speak as part of the Some prefer PaaS over IaaS clouds event at the Boston Cloud Services Meetup. Thanks J Singh for inviting me and I enjoyed speaking with many of the attendees.

Some info:

Also, for those interested, next week I am giving an extended version of this talk where there will be more time (60-75 minutes) – and I promise the demos will not be inhibited by screen resolution problems! This will be at the Boston Azure User Group meeting on Tuesday Jan 21 which will take place at the NERD Center at 1 Memorial Drive in Cambridge, with pizza provided (thanks to Carbonite).

Talk: Make the Cloud Less Cloudy: A Perspective for Software Development Teams: It’s all about Productivity

1 Reply

Today I gave a talk at Better Software Conference East 2013 about how the cloud impacts your development team. The talk was called “Making the Cloud Less Cloudy: A Perspective for Software Development Teams” and was heavy with short demos on making your dev team more productive, then a slightly longer look into how you can evolve your application to fully go cloud-native with some interesting patterns. All the demos showed off the Windows Azure Cloud Platform, though, as I explained, most of the techniques are general and can be used with other platforms such as Amazon Web Services (AWS).

Tweet stream: twitter.com/#bsceadc

http://bsceast.techwell.com/sme-profiles/bill-wilder

http://bsceast.techwell.com/sessions/better-software-conference-east-2013/make-cloud-less-cloudy-perspective-software-developmen

The deck doesn’t mention this explicitly, but all of my demos (and my slide presentation) were done from the cloud! Yes, I was in the room, but my laptop was remotely connected to a Windows Azure Virtual Machine running in Microsoft’s East US Windows Azure data center. It worked flawlessly. 🙂

Here’s the PowerPoint Deck:

Talk: Telemetry: Beyond Logging to Insight

Leave a reply

Today I spoke at the NYC Code Camp. My talk was Telemetry: Beyond Logging to Insight and focused on Event Tracing for Windows (ETW), ETW support in .NET 4.5, some .NET 4.5.1 additions, Semantic Logging Application Block (SLAB), Semantic Logging, and a number of other tools and ideas for using logging and other means to generate insight and answer questions. In order to allow this, “logging” needs to be structured, which ETW facilitates. In order for the structured data to make sense, developers need to be disciplined, which the Semantic Logging mindset supports.

The talk abstract and the slide deck used are both included below.

ABSTRACT

What is my application doing? This question can be difficult to answer in distributed environments such as the cloud. Parsing logs doesn’t cut it anymore. We need insight. In this talk we look at current logging approaches, contrast it with Telemetry, mix in the Semantic Logging mindset, and then use some new-fangled tools and techniques (enabled by .NET 4.5) alongside some old-school tools and techniques to see how to apply this goodness in our code. Event Tracing for Windows (ETW), the Semantic Logging Application Block, and several other tools and technologies will play a role.

DECK

Telemetry with Event Tracing for Windows (ETW), EventSource, and Semantic Logging Application Block (SLAB) — NYC CC — 14-September-2013 — Bill Wilder (blog.codingoutloud.com)

Talk (Guest Speaker at BU): Architecting to be Cloud Native – On Windows Azure or Otherwise

Leave a reply

Tonight I had the honor of being a guest lecturer at a Boston University graduate cloud computing class – BU MET CS755, Cloud Computing, taught by Dino Konstantopoulos.

The theme of my talk was Architecting to be Cloud Native – On Windows Azure or Otherwise. The slide deck I used is included below.

Night class is tough. Thanks for a warm reception – so congratulations and many thanks to those of you able to stay awake until 9:00 PM (!).

clip_image001.png I hope to see all of you at future Boston Azure events – to get announcements, Join our Meetup Group. We are also the world’s first/oldest Azure User Group. Here are a couple of upcoming events:

Feel free to reach out with any questions (twitter (@codingoutloud) or  email (codingoutloud at gmail)) — especially if it will be “on the midterm” – and good luck in the cloud!

Bill Wilder

book-cover-medium.jpg

Talk: Azure Best Practices – How to Successfully Architect Windows Azure Apps for the Cloud

9 Replies

Webinar Registration:

  • Azure Best Practices – How to Successfully Architect Windows Azure Apps for the Cloud @ 1pm ET on 13-March-2013
  • VIEW RECORDING HERE: http://bit.ly/ZzQDDW 

Abstract:

Discover how you can successfully architect Windows Azure-based applications to avoid and mitigate performance and reliability issues with our live webinar
Microsoft’s Windows Azure cloud offerings provide you with the ability to build and deliver a powerful cloud-based application in a fraction of the time and cost of traditional on-premise approaches.  So what’s the problem? Tried-and-true traditional architectural concepts don’t apply when it comes to cloud-native applications. Building cloud-based applications must factor in answers to such questions as:

  • How to scale?
  • How to overcome failure?
  • How to build a manageable system?
  • How to minimize monthly bills from cloud vendors?

During this webinar, we will examine why cloud-based applications must be architected differently from that of traditional applications, and break down key architectural patterns that truly unlock cloud benefits. Items of discussion include:

  • Architecting for success in the cloud
  • Getting the right architecture and scalability
  • Auto-scaling in Azure and other cloud architecture patterns

If you want to avoid long nights, help-desk calls, frustrated business owners and end-users, then don’t miss this webinar or your chance to learn how to deliver highly-scalable, high-performance cloud applications.

Deck:

Book:

The core ideas were drawn from my Cloud Architecture Patterns (O’Reilly Media, 2012) book:

book-cover-medium.jpg

Hosted by Dell:

image

Azure Cloud Storage Improvements Hit the Target

5 Replies

Windows Azure Storage (WAS)

Brad Calder SOSP talk from http://www.youtube.com/watch?v=QnYdbQO0yj4

Brad Calder delivering SOSP talk

Since its initial release, Windows Azure has offered a storage service known as Windows Azure Storage (WAS). According to the SOSP paper and related talk published by the team (led by Brad Calder), WAS is architected to be a “Highly Available Cloud Storage Service with Strong Consistency.” Part of being highly availably is keeping your data safe and accessible. The SOSP paper mentions that the WAS service retains three copies of every stored byte, and (announced a few months before the SOSP paper) another asynchronously geo-replicated trio of copies in another data center hundreds of miles away in the same geo-political region. Six copies in total.

WAS is a broad service, offering not only blob (file) storage, but also a NoSQL store and a reliable queue.

Further, all of these WAS storage offerings are strongly consistent (as opposed to other storage approaches which are sometimes eventually consistent). Again citing the SOSP paper: “Many customers want strong consistency: especially enterprise customers moving their line of business applications to the cloud.” This is because traditional data stores are strongly consistent and code needs to be specially crafted in order to handle an eventually consistent model. This simplifies moving existing code into the cloud.

The points made so far are just to establish some basic properties of this system before jumping into the real purpose of this article: performance at scale. The particular points mentioned (highly available, storage in triplicate and then geo-replicated, strong consistency, and supporting also a NoSQL database and reliable queuing features) were highlighted since they may be considered disadvantages – rich capabilities that may be considered to hamper scalability and performance. Except that they don’t hamper scalability and performance at all. Read on for details.

Performance at Scale

A couple of years ago, Nasuni benchmarked the most important public cloud vendors on how their services performed on cloud file storage at scale (using workloads modeled after those observed from real world business scenarios). Among the public clouds tested were Windows Azure Storage (though only the blob/file storage aspect was considered), Amazon S3 (an eventually consistent file store), and a couple of others.

In the first published result in 2011, Nasuni declared Amazon S3 the overall winner, prevailing over Windows Azure Storage and others, though WAS fininshed ahead of Amazon in some of the tests. At the time of these tests, WAS was running on its first-generation network architecture and supported capacity as described in the team’s published scalability targets from mid-2010.

In 2012, Microsoft network engineers were busy implementing a new data center network design they are calling Quantum 10 (or Q10 for short). The original network design was hierarchical, but the Q10 design is flat (and uses other improvements like SSD for journaling). The end result of this dramatic redesign is that WAS-based network storage is much faster, more scalable, and as robust as ever. The corresponding Q10 scalability targets were published in November 2012 and show substantial advances. EDIT: the information on scalability targets and related factors is kept up to date in official documentation here.

Q10 was implemented during 2012 and apparently was in place before Nasuni ran its updated benchmarks between November 2012 and January 2013. With its fancy new network design in place, WAS really shined. While the results in 2011 were close, with Amazon S3 being the overall winner, in 2012 the results were a blowout, with Windows Azure Storage being declared the winner, sweeping all other contenders across the three categories.

“This year, our tests revealed that Microsoft Azure Blob Storage has taken a significant step ahead of last year’s leader, Amazon S3, to take the top spot. Across three primary tests (performance, scalability and stability), Microsoft emerged as a top performer in every category.” -Nusani Report

The Nasuni report goes on to mention that “the technology [Microsoft] are providing to the market is second to none.”

Reliability

One aspect of the report I found very interesting was in the error rates. For several of the vendors (including Amazon, Google, and Azure), Nasuni reported not a single error was detected during 100 million write attempts. And Microsoft stood alone for the read tests: “During read attempts, only Microsoft resulted in no errors.” In my book, I write about the Busy Signal Pattern which is needed whenever transient failures result during attempts to access a cloud service. The scenario described in the book showed the number of retries needed when I uploaded about four million files. Of course, the Busy Signal Pattern will still be needed for storage access and other services – not all transient failures can be eliminated from multitenant cloud services running on commodity hardware served over the public internet – and while this is not a guarantee there won’t be any, it does bode well for improvements in throughput and user experience.

And while it’s always been the case you can trust WAS for HA, these days it is very hard to find any reason – certainly not peformance or scalability – to not consider Windows Azure Storage. Further, WAS, S3, and Google Storage all have similar pricing (already low – and trending towards even lower prices) – and Azure, Google, and Amazon have the same SLAs for storage.

References

Note that the Nasuni report was published February 19, 2013 on the Nasuni blog and is available from their web site, though is gated, requiring that you fill out a contact form for access. The link is here: http://www.nasuni.com/blog/193-comparing_cloud_storage_providers_in

Other related articles of interest:

  1. Windows Azure beats the competition in cloud speed test – Oct 7, 2011 – http://yossidahan.wordpress.com/2011/10/07/windows-azure-beats-the-competition-in-cloud-speed-test/
  2. Amazon bests Microsoft, all other contenders in cloud storage test – Dec 12, 2011 –
  3. Only Six Cloud Storage Providers Pass Nasuni Stress Tests for Performance, Stability, Availability and Scalability – Dec 11, 2011 – http://www.nasuni.com/news/press_releases/46-only_six_cloud_storage_providers_pass_nasuni_stress
  4. Dec 3, 2012 – http://www.networkworld.com/news/2012/120312-argument-cloud-264454.html – Cloud computing showdown: Amazon vs. Rackspace (OpenStack) vs. Microsoft vs. Google
  5. http://www.networkworld.com/news/2013/021913-azure-aws-266831.html?hpg1=bn – Feb 19, 2013 – Microsoft Azure overtakes Amazon’s cloud in performance test