Monthly Archives: May 2011

Azure FAQ: How do I run MapReduce on Windows Azure?

Q. Some cloud environments support running MapReduce. Can I do this in Windows Azure?

A. You can run MapReduce in Windows Azure. First we give some pointers, then get into some other options that might even be more useful or powerful, depending on what you are doing.

Summary of most obvious Azure-oriented choices: (1) Apache Hadoop on Azure, (2) LINQ to HPC leveraging Azure, or (3) Daytona Map/Reduce on Azure.

The first approach is to use the open source Apache Hadoop project which implements MapReduce. Details on how to run Hadoop on Azure are available on the Distributed Development Blog. Update 14-Oct-2011: Check out this write-up by Ted Kummert about his keynote at PASS where he discussed deeper Hadoop support for Windows Azure: “Microsoft makes this possible through SQL Server 2012 and through new investments to help customers manage ‘big data’, including an Apache Hadoop-based distribution for Windows Server and Windows Azure and a strategic partnership with Hortonworks. Our announcements today highlight how we enable our customers to take advantage of the cloud to better manage the ‘currency’ of their data.” Also, Avkash Chauhan provides a nice summary of the announcement.

The MapReduce tutorial on the Apache Hadoop project site explains the goal of the project, as followed by detailed steps on how to use the software.

“Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.” – from Overview section of Hadoop MapReduce tutorial

Another entrant in this Big Data Analytics space is LINQ to HPC. For more details on LINQ to HPC, check out David Chappell‘s whitepaper called Introducing LINQ to HPC: Processing Big Data on Windows. Chappell explains the value proposition, and also talks about when you might use it versus using SQL Server Parallel Data Warehouse. LINQ to HPC beta 2 is availlable for download.

[Update 19-July-2011: Daytona enters the fray] “Microsoft has developed an iterative MapReduce runtime for Windows Azure, code-named Daytona.” It is available for download as of early July, though has a non-commercial-use-only license attached to it. (credit: saw it on the insideHPC blog)

[Update 19-July-2011: It is now clear that LINQ to HPC (available in beta 2!) is supplanting DryadLINQ.] You may also be interested in checking out DryadLINQ from Microsoft Research. Though not identical to MapReduce, they describe it as “a simple, powerful, and elegant programming environment for writing large-scale data parallel applications running on large PC clusters.” As of this writing it was not licensed for commercial use, but was available under an academic use license. (With the introduction of LINQ to HPC, I can’t tell whether these projects are related, or whether LINQ to HPC is the productized version of DryadLINQ.)

And, finally, I also just read an interesting post called Hadoop is the Answer! What is the Question? by Tim Negris. This brings up some good points about the maturity of Hadoop and other points – if you are thinking about MapReduce, Hadoop, DryadLINQ, or other approaches, give his article a read.

[05-June-2011 updates] Added info from David Chappell and Tim Negris.

Is this useful? Did I leave out something interesting or get something wrong? Please let me know in the comments! Think other people might be interested? Spread the word!

Advertisements

June 2011 Azure Cloud Events in Boston Area

Are you interested in Cloud Computing generally, or specifically Cloud Computing using the Windows Azure Platform? Listed below are the upcoming Azure-related events in the Greater Boston area which you can attend in person and for FREE (or at least inexpensively).

Since this summary page is – by necessity – a point-in-time SNAPSHOT of what I see is going on, it will not necessarily be updated when event details change. So please always double-check with official event information!

Know of any more cloud events of interest to the Windows Azure community? Have any more information or corrections on the events listed? Please let us know in the comments.

They are listed in the order in which they will occur.

[10-June-2011 – added the New England ASP.NET Professionals User Group talk on June 15; I am the featured speaker. Moved Kyle Quest’s cloud hackathon to new date: June 16.]

1. 24 Hours in the Cloud – Cloud Scalability Patterns for the Windows Azure Platform

Note: GITCA’s 24 Hours in the Cloud event begins on Wed June 1 and ends on Thu June 2. This post just highlights the talk I am giving. There are MANY OTHER talks you may wish to check out. Many of the talks are IT Pro-oriented.

  • when: Thurs June 2, 5:00 – 6:00 AM (yes, in the MORNING, Boston time) [changed to earlier still! I was rescheduled to begin at 5:00 AM!]
  • where: Online – see below for registration
  • cost: Free
  • what: Talk on scalability patterns that are important for cloud applications; my session consists of a 40 minute (pre-recorded) talk, followed by 20 minutes of live Q&A. Since the talks are pre-recorded, speakers will be able to respond to questions from Twitter during the talk (then again in the live Q&A at the end) via the #24HitC hashtag. My twitter handle is @codingoutloud.
  • more info & Register: http://sp.gitca.org/sites/24hours
  • twitter: #24HitC

2. CloudCamp Boston

3. Beantown .NET Meeting – Architecture Patterns for Scalability and Reliability

4. Hack the Cloud – Cloud Platform Bake-Off

Moved to June 16th – see below

5. New Hampshire Code Camp – Concord, NH

  • when: Sat 04-June-2011, 8:00 – 4:00 PM
  • where: New Hampshire Technical Institute 31 College Drive Concord, NH 03301
  • wifi: not sure
  • food: I think they do dinner afterwards
  • cost: FREE
  • what: In the Code Camp spirit, come learn many things from many people!
  • more info: here or at www.nhdn.com
  • register: here
  • twitter: (not sure)

6. The Architect Factory

  • when: Thu 09-June-2011, 1:00 – 8:00 PM
  • where: Hosted at NERD Center
  • wifi: Wireless Internet access will be available
  • food: (not sure of details yet)
  • cost: FREE
  • what: Real, Practical Guidance on becoming and Architect, or becoming a Better Architect
  • more info: http://architectfactory.com/
  • register: http://architectfactory-eorg.eventbrite.com/
  • twitter: #architectfactory or #af3 (not sure which is “official”)

7. New England ASP.NET Professionals Group

  • when: Wed 25-June-2011, 6:15 – 8:30 PM
  • where: Microsoft Office on Jones Road, Waltham
  • wifi: no
  • food: group does to dinner afterwards
  • cost: FREE
  • what: A talk that introduces Cloud Computing and the Windows Azure Platform and shows how it relates to the ASP.NET developer – tools, libraries, and how to build and deploy.
  • more info: http://neasp.net/
  • register: see http://neasp.net/
  • twitter: (not sure)

8. Hack the Cloud – Cloud Platform Bake-Off

9. 4th Annual Hartford Code Camp

  • when: Sat 18-June-2011, 8:00 – 5:30 PM
  • where: Hosted at New Horizons Learning Center (Bloomfield CT)
  • wifi: Wireless Internet access will be available
  • food: Pizza and drinks will be provided
  • cost: FREE
  • what: In the Code Camp spirit, come learn many things from many people!
  • more info: http://ctdotnet.org/
  • register: see http://ctdotnet.org/ until a direct link is published
  • twitter: (not sure)

10. Boston Azure User Group meeting: Rock, Paper, Azure Event!

  • when: Thu 23-June-2011, 6:00 – 8:30 PM (come at 5:30 if you need help getting set up)
  • where: Hosted at NERD Center
  • wifi: Wireless Internet access will be available
  • food: Pizza and drinks will be provided
  • cost: FREE
  • what: Bring your Windows Azure-ready laptop (or get a loaner, or pair up with someone) as we go head-to-head in an Azure programming contest (it is a simple game, but you will compete with others in the room). Also, there will be prizes – like an Xbox 360, Kinect, and other goodies.
  • more info: See Boston Azure cloud user group site for details or see Jim O’Neil’s blog post on the event. Of special note is to request your free account (no credit card, etc. – easy), following details on Jim’s post – takes only a minute of YOUR time, but will help make sure you don’t need to wait for it on Thurs night – do it NOW! 🙂
  • register: here
  • twitter: #bostonazure

11. Maine Bytes: Azure State of the Union

  • when: Thu 30-June-2011, 6:00 – 8:00 PM
  • where: Unum’s Home Office 3 building at 2211 Congress Street in Maine
  • wifi:
  • food:
  • cost: FREE
  • what: Ben Day will give a talk: “Microsoft’s Azure platform moves fast and new features get added all the time. It can definitely be tough to keep up. In this session, Ben will give you a tour around the current features and offerings in Azure with some tips on how to use them in your applications and how to integrate Azure into your software development process.”
  • more info: See http://www.mainebytes.org/ for details
  • register:
  • twitter:

Coming in July:

  • Boston Azure User Group meeting on July 28
  • And more? Please let me know in the comments if you know about an event relevant to those who care about the Windows Azure Platform

Omissions? Corrections? Comments? Please leave a comment or reply on the Twitters!

Spoke about Azure and Cloud at New Hampshire .NET User Group

I was the guest speaker at the May 25th meeting of the New Hampshire .NET (NHDN) user group in Concord, NH at the New Hampshire Technical Institute.

Here is the slide deck I used for the talk: Demystifying Cloud Computing and the Windows Azure Platform.

Scaling Cloud Applications – Birds of a Feather (“BOF”) Session at TechEd

Today at TechEd in Atlanta, I served as discussion leader for a Birds of a Feather (BOF) session on Scaling Cloud Applications. The session had around 20 people in the room, and an unknown number watching the live stream, some of whom actively participated over Twitter.

"Bird's Nest" Panel

Some of the topics discussed:

  • SaaS vs. PaaS vs. IaaS, including the blurring of the lines between them
  • Scale Up vs. Scale Out vs. Scale back down — elastic scale means you pay for what you use — just start (or stop) using the resources you need and the billing will reflect this usage
  • Scale has many dimensions, some of which are Geographic Distribution of and Number of Users, Amount of Data, and Needed Computation Power
  • Cloud applications are architected differently, often decoupling user-facing functionality from services – the front-end may communicate with the back-end using a reliable queue (such as offered by Windows Azure); see CQRS pattern
  • There are many application architecture concepts that are shared across applications built for most cloud vendors – for example, the loosely coupled front-end/queue/back-end scenario mentioned above can be implemented on Windows Azure (which provides Web Roles, reliable queues, and Worker Roles) or Amazon (which allows you to build and upload a Virtual Machines for front-end and back-ends, plus offers a reliable queuing service), other cloud platforms, and even on-premise – the cloud services just make these more natural to implement
  • Improving latency for cloud applications might be facilitated through a Content Delivery Network (CDN), geographic load balancing (such as through Windows Azure Traffic Manager), and other techniques

Many thanks to all who participated, including:

The BOF events were very well run by the INETA team (Chris Pels and a few others).

If you want a more structured treatment of some of the same scalability concepts, feel free to check out my talk on Cloud Scalability Patterns coming on June 1/June 2 via the GITCA “24 Hours in the Cloud” event. Here is the generic event description – follow the link to find out when my talk is slotted in.

Come and get your Cloud geek on! On 1st June, 2011 GITCA and Microsoft are running an event called “24 Hours in the Cloud”. There will be 24 one hour sessions around the world covering a wide range of Cloud Computing topics. The presenters will be live on twitter to answer your questions. I will be one among them. There is something for everybody, developer, IT pro and SQL enthusiast. There is no question that Cloud Computing is here to stay and this is a unique opportunity to be educated and gain an insight as to where Cloud Computing is going. Stay tuned for more details, such as how to join the “Cloud 24 hour party”, as the event date approaches.

If you have more questions on the topic, feel free to put in on twitter (@codingoutloud), you can comment on this blog post, or you can email me (coding out loud at gmail). And, finally, below you can find the  Twitter stream from the live event – latest on top, earliest on bottom – (which I salvaged via research.ly). Scale on!

@techedbof20117 hrs ago
BOF12-DEV on Cloud Computing is now coming to an end. #bofdev #msteched

@techedbof20118 hrs ago
Mobile apps are a big area for growth in the cloud computing area.
#bofdev #msteched

@techedbof20118 hrs ago
Talking about moving existing apps to the cloud as we near the session conclusion.
#bofdev #msteched

@techedbof20118 hrs ago
When you scale you can select the instance size in Azure. #bofdev #msteched

@techedbof20118 hrs ago
Decoupling front end from back end processing is an important concept. #bofdev
#msteched

@techedbof20118 hrs ago
For scaling, one experience is knowing how much work to do and
how much an instance can process in an hour. #bofdev #msteched

@techedbof20118 hrs ago
Would be nice if the Azure platform would monitor and scale for you.
#bofdev #msteched

@techedbof20118 hrs ago
@itagsubbu Great to have you join us. #bofdev #msteched

@itagsubbu8 hrs ago
@techedbof2011 Thanks for asking my question. I am watching this live event.
#bofdev #msteched

@techedbof20118 hrs ago
@itagsubbu Yes you can scale either on a scheduled basis or in a programmatically
#bofdev #msteched

@itagsubbu8 hrs ago
@techedbof2011 #bofdev #msteched Can we scale up for certain period in an year?

@rileybeebs8 hrs ago
RT @jmilgram: Getting ready to attend Bill Wilder @codingoutloud Designing Scalable
Cloud Applications #bofdev session at TechEd #mstech ...

@techedbof20118 hrs ago
Gmail is an example of SaaS #bofdev #msteched

@techedbof20118 hrs ago
How do you get resources to the cloud platform? #bofdev #msteched

@techedbof20118 hrs ago
IaaS eliminates the infrastructure but you admin, for PaaS both roles are eliminated
#bofdev

@techedbof20118 hrs ago
As you scale out across geographically dispersed data centers what is the impact on
SQL Aszure costs? #bofdev

@jimoneil8 hrs ago
blurring of IaaS and PaaS is something interesting as well... becoming less of a
differentiator? #bofdev

@techedbof20119 hrs ago
Silverlight app has 50K updates/sec #bofdev

@techedbof20119 hrs ago
Silverlight app that was not designed for the cloud. What to do? #bofdev

@techedbof20119 hrs ago
Thoughts on WPF app w/ Azure backend? Are you doing that? #bofdev #msteched

@techedbof20119 hrs ago
BOF12-DEV on designing scalable cloud applications is getting started
#bofdev #msteched

@TashasEv9 hrs ago
RT @jmilgram: Getting ready to attend Bill Wilder @codingoutloud Designing Scalable
Cloud Applications #bofdev session at TechEd #mstech

@TashasEv9 hrs ago
@rileybeebs I haven't forgotten about you! just haven't been able to leave the
#BOFDEV sessions at all yet!

@TashasEv9 hrs ago
The next #MSTechEd #BOFDEV : Designing Scalable Cloud Applications lead by
@codingoutloud

@techedbof20119 hrs ago
The next #MSTechEd #BOFDEV : Designing Scalable Cloud Applications
lead by @codingoutloud

@techedbof20119 hrs ago
RT @jimoneil: RT @codingoutloud my 1:30 TechEd session on Designing Scalable
Cloud applications #msteched #bofdev << will be heckling from afar!

@jimoneil9 hrs ago
RT @codingoutloud my 1:30 TechEd session on Designing Scalable Cloud applications
#msteched #bofdev << will be heckling from afar!

Join the TechEd Windows Phone 7 Unleashed Hackathon

Windows Azure + Windows Phone 7 = Better Together!

Find out why on Monday May 16 @ TechEd in Atlanta – at the FREE After-Hours Dinner-Included Hackathon!

Windows Phone 7 Unleashed Hackathon
Monday, May 16, 2011
6:00 PM to 11:00 PM
Register: http://bit.ly/RegWP7Hackathon

Don’t miss this opportunity to get hands-on help with your Windows Phone 7 app, from the experts!

This is a “hands on” hackathon where you will learn from Windows Phone 7, XNA and Azure experts how to build, scale and publish your Windows Phone 7 app or game. If you are just a beginner, or already have apps in the Marketplace this event should not be missed.

Come hear about new developments that will help you combine the power of the Windows Phone with the Windows Azure cloud platform.

BYO Laptop! And prepare yourself for an energetic evening of fun, learning, and accomplishment!

RSVP early, as space is limited to 300 attendees: http://bit.ly/RegWP7Hackathon

Food, beverages and refreshments will be provided.

Cure for “NO INSTALLATION MEDIA” Error when Zune Installer Can’t Find the Media for Installation Package

How I got around Zune’s “NO INSTALLATION MEDIA” and “Can’t Find the Media for Installation Package” error

 I recently reinstalled Windows 7 on one of my computers and in rebuilding my development tool set, including for Windows Phone, and found I could not run a Windows Phone 7 project locally: Visual Studio complained I did not have the Zune software installed. Okay, not a problem; I will install Zune. But not so fast…

I encountered the following mysterious error while trying to install the Zune software to my Windows 7 desktop.

What does this Zune error message mean?

 
Looking at the text of the message did not help me or yield obvious clues:

NO INSTALLATION MEDIA

Can’t find the media for installation package ‘Windows Media Format SDK’. It might be incomplete or corrupt.

Error code: 0x80070002

Searching around the internets did not help, though I saw a reference to do a few things, one of which was to install the latest Windows Media Player. Well… it turns out, I had NO version of the Windows Media Player installed, so I simply installed the latest, and the Zune installer was happy…

One more step

But Visual Studio 2010 was NOT yet willing to allow me to run the Windows Phone 7 emulator to test and debug my Windows Phone applications. I saw the following additional (but improved!) errors from Visual Studio.

First, could not deploy. Nothing new here:

But the reason provided looked more promising:

This is a better known error, easily rectified. Simply switch to the emulator if your project is referencing an attached device, done at the top of Visual Studio as shown here:

Okay… now back to Windows Phone 7 development – of course, with a Windows Azure back-end using the Windows Azure Toolkit for Windows Phone 7.

New England Code Camp 15

I attended New England Code Camp 15 today and attended a bunch of interesting talks, and I also gave a couple of talks myself. (Links to my slide decks are included below.)

At my talks, I mentioned the Windows Azure Pass – a 30 day FREE pass for using Windows Azure Compute (IIS or Worker Roles), SQL Azure, Azure Blobs/Tables/Queues, etc. If you didn’t get a handout at talk, no worries! – You can still access the offer: Go here and use Promo Code BILLONAZURE. Let me know if you have any questions or if you use the promotion.

Talks I attended:

  • Maura Wilder and Joan Wortman‘s talk on the Ext JS JavaScript framework (which I learned has an incredibly rich widget library and robust  programming model).
  • Richard‘s talk on becoming a better developer.
  • Ben Day‘s talk on 7 Lessons Learned during his first large Silverlight dev project. Find out more by reading Ben’s article on same topic, starting here.
  • Steve Maier‘s talk on using Azure-hosted WCF services to serve as your mobile application’s back-end.
  • Chris Bowen on HTML 5.

My presentations (including links to the PowerPoint slide decks):

Many thanks to Chris Pels, Chris Bowen, and especially Patrick Hynes for such a great event! Thanks also to Telerik and Wintellect for sponsoring our food!

Also enjoying hanging out afterwards at Uno’s with Maura, Joan, George Babey, John Garland, Jesse Liberty, Pat Tormey, Chris, Veronica and Shawn Robichaud, Ron, and several other folks I didn’t get to say hello to…