Cloud Confusion and the DoD

Last week the Department of Defense announced a new private cloud computing environment called RACE (Rapid Access Computing Environment). This was an announcement that received a ton of attention in blogs and the circles of cloud computing advocates and naysayers. The frustrating thing about announcements and the ensuing arguments that take place afterward is that there are very few people that really understand what cloud computing is and all of the facets that are involved. The problem really stems from the over-use of the term "cloud computing" to cover what seems to be every application that runs in your browser. Just because you don't know where the actual server is located, or because it runs in your browser does not make something cloud computing. As soon as the announcement was made about RACE, the comparisons to Google came flying in. First of all, this is not even the right comparison. I am behind using cloud computing for government applications, but if we are going to compare the systems, let's at least compare them fairly. RACE is more of an IaaS cloud service and Google is both SaaS (GMail, Google Docs, etc) and PaaS (AppEngine). It seems more fairly compared to the single service of EC2 provided by Amazon. I would love to take a deep dive into RACE, but unfortunately, and as expected for security, it is only available to someone with a government card or clearance. What they do say on the website though is that provisioning after a request only takes 24 hours. Is that truly on demand computing? Call me spoiled being able to provision dozens of servers with a single web service request in 5 to 10 minutes, but that is what I expect from a system now that claims to be on-demand. This is a great direction and good start for DISA. It is very encouraging to see the activity in the government space around cloud computing. We really need to stop lumping every cloud computing service or offering under the single banner of Cloud Computing. People will continue to make their offerings sound better than others with unfair comparisons and statistics. A common language and standard is needed for fair comparison. Until then, let the spin go on!

Why Social is Successful

Almost every time that I speak, I can count on being asked about where the internet is going and what technology will be the next "big thing". Obviously there are a ton of people that would capitalize on that if anyone could predict the future. Entrepreneurs, venture capitalists and angel investors spend their days trying to decipher and figure out the answer to that question. It is clearly a tough question.I love technology and the constant curve of innovation that surrounds me at work everyday. With the new hot technologies being social applications, the whole world is getting a little smaller still. It is not surprising to me that applications such as Twitter and Facebook are so wildly popular. It does help that they are free, but the "freemium" model works like a catalyst. People generally don't sign up for something just because it is free, but because it is useful. These applications are successful because they are relational and they bring us into closer contact with one another. The internet a great warehouse of all sorts of information at your fingertips, but that only satisfies our curiosity and natural thirst for knowledge. Most people are looking for more meaningful relationships and someone that cares. The corporate website that only displays information and does not allow for a round trip of communication does not "get it" yet. That is the old way of doing business on the internet. Simply having a presence is not good enough. The internet has evolved to allow us what we really want: a conversation, and it will continue to move in that direction. The first generation of internet technology satiated our hunger for knowledge and now it is helping us connect with others in more meaningful ways. Social is sexy because that is how God made us. Just look at what the top search terms are on a daily basis: they are for people, celebrity figures, that we want to know about. It is really, a desire to get to know them better. It is the next best thing to a conversation. Dating sites have been wildly popular for years, and now, the feeling of instant connection with Facebook and Twitter are where we feel comfortable. We are able to connect with people we haven't seen in years and people we have never met. We can share our life experiences and our knowledge across all boundaries and all over the globe. The next technologies on the internet will bring us even closer and more connected. The written letter was and is an excellent form of communication, but it was made faster with the telegram, the telephone, and the cellphone - continually bringing us closer in relationships with greater and greater ease. I believe that internet technology will continue to do the same. Whether it allows us to take a course online from a renown instructor, get us just a little closer to our family, or connects us better with a business contact, social applications (done well) will continue to be successful. As to what it will be... well, let's talk about it on Twitter.

Azure verses Amazon AWS Pricing

Microsoft released the pricing today for their cloud computing platform called Azure.  The costs really look competitive when compared across the other platforms that are currently offered in the market.  The breakdown is as follows:

• Windows Azure
• Compute = $0.12 / hour
• Storage = $0.15 / GB stored / month
• Storage Transactions = $0.01 / 10,000
• Bandwidth = $0.10 IN / $0.15 OUT / GB
• SQL Azure
• Web Edition - Up to 1 GB relational database = $9.99
• Business Edition - Up to 10 GB relational database = $99.99
• Bandwidth = $0.10 IN / $0.15 OUT / GB
• .Net Services
• Messages = $0.15 / 100,000 message operations
• Bandwidth = $0.10 IN / $0.15 OUT / GB

As you can see, they look fairly competitive to Amazons pricing in the different areas which can be found on the pages for EC2, S3 and SQS.  It is very hard, if not impossible, to do a complete side-by-side comparison that covers all of what is offered and all of the different configurations and software stacks, but let's look at some of the basics to get an idea.

Let's first look at the storage services offered by the platforms since they are almost identical in function and feature.  Azure's platform for storage is a flat $0.15 / GB / month and
$0.01 / 10,000 transactions.  Data transfer across all of Azure's services is $0.10 IN and $0.15 OUT / GB. This makes the calculation for storage very simple on their part.  S3 has a tiered pricing scheme which makes it a little more difficult to calculate, but does offer some cost saves when storing vast amounts of data.  On light storage (1-50 TB) Amazon will charge $0.15 / GB for storage, $0.10 IN and $0.17 OUT / GB in bandwidth.  Amazon divides the types of transactions in their pricing: $0.01 per 1,000 PUT, COPY, POST and LIST request and $0.01 per 10,000 GET (and other) request.  This ultimately gives Azure the win on the low end for storage.  As soon as you go over the 50 TB / month threshold for storage or transfer more than 10 TB of data OUT of storage however, Amazon wins in price on the growth.  The more your store, the less it will cost you on Amazon, even going down to $0.12 / GB.  The data transfer rates going out can get as low as $0.10 / GB OUT.  Amazon also does not charge you for data transfer from S3 to EC2 (inside the cloud).  It is not clear if the same holds true for Azure. So, if you have a lot of data, then Amazon will give you the better prices, but if you are light on data and traffic, then Azure has the best price.  All that being said, it is important to note that Azure is severely limited during the preview period to 50 GB of storage.  They could add tiered pricing in the future to compete with Amazon's pricing at larger volume.

Probably one of the most anticipated, but hardest thing to compare, was the pricing on the compute hours for Azure.  This is extremely difficult for many reasons: Amazon offers a dozen different types of operating systems while Azure is the only environment.  Amazon virtualizes the memory and cpu so you have the option of adding horsepower at added costs and Azure has not published any information about the underlying metrics of a computational unit. Azure takes care of all of the scaling and load balancing within the fabric and these are add on services that you must pay for with Amazon.  Lastly there is an upper limit of 2000 VM hours right now on Azure during the preview period.  With all that being considered, if you compare the lowest price Windows installation on Amazon at $0.125 / hour to the $0.12 / hour on Azure, then Azure wins on price ever so slightly.  It is very hard to say how the "speed" compares on the two systems by virtual core and if this is even a fair price comparison, but it is the closest we can get for a comparison.  When included the tiered pricing from Amazon for data transfer, which lowers the cost per GB as you go up, and include Amazon's costs for Auto Scaling, and Elastic Load Balancing then the waters start to get very muddy.

The easiest to compare is the messaging systems on both cloud offerings.  Azure is offering 100,000 message operations for $0.15, plus the regular $0.10 IN and $0.15 OUT / GB.  Amazon offers 100,000 message operations for $0.10, plus the Amazon tiered pricing for data.  It is very likely that the majority of the messaging operations on these platforms will stay internal to their platform which takes the cost on Amazon for data virtually down to $0.00.  In this case Amazon beats the Azure pricing by 1/3.

I was happy to see Microsoft release the pricing for Azure today as I have been anxious to see what the cost comparison would be.  With pricing relatively close between the two, I would venture to say that Azures success will depend on its ease of use and streamlined single platform while Amazon will continue to thrive on its openness and plethora of options.

North Carolina and the Amazon Associates Program

The North Carolina General Assembly has a mandate to balance the budget every year, which I can appreciate because at least it keeps us somewhat focused as a state. I guess I have a very hard time understanding how the General Assembly can try to get that money out of companies that are not even located within our state borders.
I am a member of the Amazon Associates program which will pay a small commission on products sold from simply advertising on your site.  I have not made any significant amount of money from this, but there are several people in our state that do very well supplementing their income and even forming a small business out of concepts such as the Amazon Associates Program.  The North Carolina General Assembly has now added to the budget that Amazon must pay the taxes on products sold through the program to North Carolina.  Amazon's answer.... pull the plug and don't offer it anymore.  I don't blame them.  What services is North Carolina providing for them?  They do not have any buildings here that our Fire Departments, Police Departments or Emergency services help protect.  They don't have any cars or trucks that use our streets and highways.  They do not have a physical presence here that consume any of the state provided services.  Why should they pay taxes?  What the General Assembly is doing is stifling small business and is really doing absolutely no good for the state.  It only serves to drive companies away from our state.  After passing a budget with this nonsense in it, why would Amazon even consider placing a facility here or locating a division here within North Carolina.  The passage of this item buried in the middle of the budget has produced no additional revenue for North Carolina and only serves to drive business away and make its own citizens consider moving to neighboring states.  Here is an excerpt from the email that I received from Amazon explaining the shut down of the program within North Carolina:

We are writing from the Amazon Associates Program to notify you that your Associates account has been closed as of June 26, 2009. This is a direct result of the unconstitutional tax collection scheme expected to be passed any day now by the North Carolina state legislature (the General Assembly) and signed by the governor. As a result, we will no longer pay any referral fees for customers referred to Amazon.com or Endless.com after June 26. We were forced to take this unfortunate action in anticipation of actual enactment because of uncertainties surrounding the legislation’s effective date.

If we are going to look for ways to balance the budget, let's make sure that we actually have the ability to collect money or cut the money from many of the wasteful programs already in place.  Please don't drive business and people away.

CloudFront Added to AWS Console

The Amazon Web Services management console has a new addition. A new tab appeared this morning at the top of the console at http://console.aws.amazon.com allowing you to manage your CloudFront distributions.

Using the console, you can choose to create a new distribution, select the bucket from within S3 and add up to 10 CNAME entries for the distribution. This functionality is convenient, but is not the functionality that I would have liked to see added to the console next as you have been able to do all of this with S3Fox for several months now.  I was hoping to see the management functions for CloudWatch, Elastic Load Balancing and Auto Scaling to appear in the console.  These should be available in the console this year I have been told, but it was wishful thinking to expect them this early in the year.

DigitalChalk at ASTD

DigitalChalk is going to be giving away $300 over Twitter at the ASTD 2009 Conference in Washington DC this year.  If you are at the conference and have a Twitter account, you can play.  Watch crazy man Josh in this video...



I heard that they were at the White House yesterday and President Obama asked how he could play, but they sadly had to turn him down since he isn't going to be present at the conference.  Oh well, maybe next time.

The goal of the game will be to figure out a word or phrase that will be on the back of a bunch of shirts running around the conference.  If you want to be a live participant in the game, go see Josh and Tony at booth 1519 and make sure that you follow DigitalChalk's Twitter account http://twitter.com/digitalchalk .  More information is available on Tony's blog.

In the Works at DigitalChalk

There are a lot of exciting things happening at DigitalChalk right now.  The development, operations, and quality teams are very hard at work on a new release of the product coming very soon.  Code named "Einstein", this release is packed full of feature requests from our customers.  While we wrap up the open tickets and complete QA, I thought I would start giving a sneak peak at some of the features that are going to be included by writing about them here in my blog over the next couple of weeks.

We have done a lot of work on the interface of DigitalChalk in the Einstein release.  Immediately you will notice that we have changed the look to be much more streamlined.

We had a couple of goals in moving this direction.  First of all, we really wanted to be able to provide much more information on a single page to the instructor or student.  This is really a challenge because we needed to balance between a page that feels cluttered and has too much information and keeping it "too clean" where the information cannot be found on the same page without navigating away.  It is also important to us that Einstein is compatible with a wide range of browsers.  Those of you that have done any development at all on the web know what a pain that can be.  Something that works in Internet Explorer will not work in Safari, and something that works in Firefox will not work in Internet Explorer and that isn't even taking into account all of the different versions of the browsers.  Sometimes this feels like a loosing battle for a complicated site.  Page rendering time has also been an area that we have devoted significant resources toward.  We are seeing an improvement now of over 500% on some pages!  I will be including more screenshots in the future of various parts of the site as I talk about specific changes.  We are all very excited about the changes and can't wait to push them out to you.

Apple to Build Data Center in NC?

News hit the street this weekend that Apple could be considering North Carolina as its next data center location. Just a couple of years ago Google selected Lenior, North Carolina as a location for a $600 million dollar data center and Apple may be joining them in the Tarheel state. The story is that the North Carolina legislature is offering large tax breaks to Apple in order to attract them to the area. I am happy to see that we are starting to think a little more about the types of jobs that will sustain the economy in the future. Computing power will always be needed and the demand for it is every growing. Technology will continue to drive much of the innovation that is occurring today and I welcome more of the support of that here in my home state. It has been sad to see the textile and furniture industry cause so much job loss and heartache as it has moved elsewhere and overseas, but it is time for us to look to the future and continue to reinvent ourselves. Another data center will drive more need for bandwidth and reliable power and will continue to draw more technology jobs this direction. It would thrill me to see North Carolina, especially Western North Carolina, become the Silicon Valley of the east. We are a long way from that now, but let's push forward and look ahead. Come on Apple, we are ready for you.

Amazon Import/Export

After a recent talk that I gave on cloud computing, one of the attendees contacted me with some questions about the "safety" of the data and also wanted to talk about vendor lock-in.  It is no secret that I am a fan of the Amazon Web Services cloud platform and so it followed that these questions all had to do with the way Amazon stores the data.  While these are typical questions, today the second question became much easier to answer.  Amazon Web Services announced the availability of AWS Import/Export.  Quite simply, Amazon is offering its customers a very easy way to ship a disk of data to them and they will push it into S3 to your specifications or grab your data out of S3 and put it on the disk for you.  This is very attractive because it can take days to actually move hundreds of GB on your office network into or out of S3 because of the simple limitations of bandwidth at the average workplace.  I am not a fan of vendor lock-in and have thought long and hard how to avoid it.  Any code that we write storing to and from Amazon's infrastructure is isolated enough so that we can switch it to another provider by reimplementing that single area. But, the fact still remains that it would take weeks to move all of our data and it would have to be done over an extended period of time. A strategy for this is to move data between two cloud providers instead of bringing it down locally and putting it back up. While moving over a backbone is still going to be faster, it will not be optimal.  With AWS Import/Export you are one step closer.  For $80.00 and $2.49 per hour, you can currently have all of the data on a disk pushed into Amazon's cloud and the export facility is coming soon.  I commend Amazon for providing so many tools and conveniences.

Amazon Releases New Cloud Computing Services

One of the big draws to the cloud is its ability to scale with your application.  That has become much easier with Amazon Web Services today.  Early this morning the Amazon Web Services team launched three new services: CloudWatch, Auto Scaling and Elastic Load Balancing.  Combining the use of these three services allows a user to configure and scale their application based upon information gathered by CloudWatch.  These are important additions to the Amazon Web Services offerings because it helps take more of the coding and configuration work away from the developers and system administrators.  This is a key benefit of cloud computing that AppEngine from Google and Azure from Microsoft have built in and they have kept completely transparent to the developer.  While Amazon has not made it completely transparent with the release of these services, it is a great step and may be exactly the middle ground that is needed.
CloudWatch allows you to monitor CPU utilization, data transfer and disk usage, request rate and traffic to your EC2 instances.  Based up on the information that CloudWatch gathers, you can set triggers that will look at that data over a time period and allow you to use the Auto Scaling to automatically add or remove EC2 instances to the specific group of machines working on a particular task.  Finally the Elastic Load Balancing helps you distribute the traffic coming into your application to your EC2 instances.  This is a welcome addition as it accomplishes fault tolerant load balancing for us without the cost of having to setup several HAProxy instances.  So, even though we incur the costs of using the new Elastic Load Balancing Service, it quickly pays for itself because we are able to remove our own load balancing configuration on EC2.  I am excited to see these new services finally go beta to the public and I am looking forward to more.

SpringSource Aquires Hyperic

It certainly isn't as big of a story as Sun being acquired by Oracle, but it is worth noting.  SpringSource has announced that they have acquired Hyperic.  SpringSource is the company that is the driving force behind the Spring Framework which is arguably the most widely used open source Java framework in enterprise software today.  Hyperic provides a software suite for monitoring applications and servers and they have recently been dabbling in providing some of these services for the cloud.  The team up of these two vendors is especially interesting because they both offer some great services and tools in through open source.  This partnership could really allow even more granular visibility into the Spring stack for monitoring.  It will be interesting to watch and see if we start to see a move to OSGi component monitoring especially in the context of the SpringSource dm Server.  I hope to see great things out of this and expect that there will be a lot of value there for the open source community.  We should watch for more tools for monitoring applications in the cloud as well from this pair.

Tips on Using Amazon CloudFront

If you are serving a lot of content with an audience that is spread out over the globe, then most likely you are in need of putting your content on a CDN (Content Delivery Network).  There are a host of options for you such as LimeLight, BrightCove and Akamai to name a few, but I have found that Amazon offers much of the same functionality at a better price.  Moving your static content to Amazon's CloudFront can really reduce the load on your servers and certainly enables you to serve your content from a place that is closer to the request.  The best thing about it is that it is very very simple to do.  There are some pieces of information that can really save you alot of time and energy if you are converting your site to serve content from CloudFront.  Here are some things that were important to me and that I had to address when moving our site:

1. Speed
• Faster delivery of all content from edge locations close to the request
• Faster load time of a page in the browser
2. Ability to serve content over HTTP and HTTPS
3. Simple, straight-forward development

Speed
Amazon CloudFront allows you to manage the content in your S3 buckets and enable them for delivery from edge locations all over the world.  To enable a bucket for CloudFront, it is a simple API call to create a Distribution.  Once your Distribution is provisioned, which usually takes less than 15 seconds in my experience, you will be given a Distribution url which will look something like: http://abcd1234.cloudfront.net .  This Distribution URL can be simply though of as another URL to that bucket in S3.  For instance, if you have a bucket in S3 named "mys3bucket" and you create a Distribution for that bucket, which returned http://abcd1234.cloudfront.net, then you could reference that bucket as http://s3.amazonaws.com/mys3bucket/ or as http://abcd1234.cloudfront.net .  The only difference between the two is that the CloudFront URL will be served from the location closest to the request and the content from that distribution can be served at 1,000 Megabits per second and even more if needed and requested.  For the easy creation of your S3 Buckets and CloudFront distributions, I suggest using S3Fox for Firefox.  With a simple right click on your bucket through S3Fox, you can create a new Distribuion in seconds.
Beyond just offering the serving of your content from an edge location at high throughput, CloudFront distributions have another advantage that can speedup the loading of your sites pages.  Using a single Distribution, you can assign up to 10 CNAMEs to it.  This can really make a large difference when serving content for a page.  Most browsers block and only allow 2-4 parallel connections to a single host when loading content.  Creating 10 CNAME entries all pointing to the same distribution allows the browser to make 10 times the parallel connections it could using a single entry.  I would suggest Steve Souders book High Performance Web Sites: Essential Knowledge for Front-End Engineers for more tips on this.

HTTP and HTTPS Delivery
One of the drawbacks to CloudFront is that it does not support HTTPS delivery of the content.  However, there is some good news here because S3 does support HTTPS.  Because the content that you need to deliver over CloudFront sits in an S3 bucket, then you can also deliver that same content over HTTPS but you cannot take advantage of the edge locations.  This is especially useful when you need to serve a secure site that has a bunch of images that you are already serving from CloudFront.  If you try to serve those images over HTTPS, your users will get a security warning or no image at all because the browser will block it.  So, when you need to serve those secure pages, simply switch your delivery URL to the S3 location instead.  A word of advice here is to create your S3 bucket without using any '.'s.  I know that this really messes with you if you want your URL to look like it is coming from your servers, but it will allow you to serve secure content from S3.  The S3 bucket is served over HTTPS using a wildcarded certificate.  This allows Amazon to serve your content using any prefix as long as it does not have any extra dots ('.').  If your bucket name is 'mys3bucket', then you can create a CNAME record that points 'mys3bucket' to mys3bucket.s3.amazonaws.com.  Because the wildcarded SSL certificate is to *.s3.amazonaws.com, then you can serve your content from https://mys3bucket.s3.amazonaws.com.  While this isn't the perfect solution, it is really nice for serving the same bucket content over HTTP or HTTPS.


Simple, Straight-Forward Development
I am not sure that building scalable applications is ever simple, but I am always looking for ways to make it extremely repeatative and simple for myself and other developers. Because we are using Java for most of our applications, the easiest way for me to do that was through a tag library.  I included all of the logic for switching between the different CNAMES for CloudFront delivery and logic for detecting a secure connection and need to switch to straight S3 delivery right in the tag.  There were some interesting details that were necessary to take full advantage of CloudFront.  One was coming up with a hash scheme for a URL so that it could be cached in the browser and distributing the calls across the CNAMEs evenly.  The second was to make sure that the tag was written so that it was at the request level instead of the page level.  This helped ensure that even an included JSP would not cause a narrowing of the number of CNAMEs that I could use.  This is the method in the tag that does most of the work

protected String getCDNUrl() {
   String result = null;
   if (((PageContext) getJspContext()).getRequest().isSecure()) { 
      result = getCDNSecureUrl();
   } else {
      result = getCDNUrls().get((1 <= getCDNCount()) ? Math.abs(getFile().hashCode()) % getCDNCount() : 0);
   }
   return result;
}

CloudFront has proven to be very useful to me.  I hope with a few of these tips that you can improve the performance of your site.  Please share your experience with me or any other tips that you might have!

Google AppEngine supports Java

I should be sound asleep resting up for a big day tomorrow, but I couldn't resist.  I saw an announcement hit Twitter that I have been waiting on for a long while.  Google's AppEngine team has announced support for Java and has also released an Eclipse plugin that makes development much easier.  They are letting the first 10,000 interested in Java into give it a test run, so go over and get signed up.  If you are running Ganymeade then you can grab the Google AppEngine plugin from the update site at http://dl.google.com/eclipse/plugin/3.4.

It was extremely simple to go through the happy path after installation and get my first GWT application up and deployed.  In fact, Google provided all of the code.  Nothing better than a full example to get you going.
After installing the plugin, you will see that there are three new buttons that have been added to the toolbar.

 

They are for creating a new web application, compiling and deploying to AppEngine.  All that you need to do is click on the first button, name your application and give it a package.  After that, simply right click on the project and choose Run As > Web Application.  This will activate your local AppEngine installation and show the application which simply asks your name and gives a confirmation.

 Congratulations!

OAuth and Twitter

 Earlier last month, Twitter released OAuth access to their site and to the API.  Excited about the possibility of integrating some DigitalChalk functions with Twitter, I decided to take a couple of hours to play around with it and see what it offered.  There are lots of examples on the Twitter API site, but none of them are in Java.  What?  Really?  Maybe because it is so easy to do, but I do think that it is worth posting about.  First, a little about the flow of getting your access credentials from Twitter.  The diagram below shows the order of events and some pieces of the data that you need to get started.

For my implementation example I am using a Java OAuth library written by John Kristian, Praveen Alavilli and Dirk Balfanz.

1. You must get your Consumer Key and Consumer Secret by registering your application on Twitter.com

2. The goal is to get an Access Token and Secret that you can use to read and/or write to a Twitter users information without having to ask for their username and password everytime.  We also don't want to have to store that information because they may change it on Twitter.com and we don't want to have to synchronize the information. (The user doesn't really want to give that information to us anyway)
   
3. The first step is get request a Request Token from Twitter.  You do this by using a timestamp, nonce, oauth version, and your consumer key and signing it with your consumer secret.  Then a request can be made to http://twitter.com/oauth/request_token including the signature.
   

public TwitterTokenPair getRequestToken(TwitterConsumerCredentials credentials) throws TwitterOAuthException {
   TwitterTokenPair result = null;
   OAuthAccessor accessor = newAccessor(credentials);
   OAuthClient client = new OAuthClient(new HttpClient3());
   try {
      client.getRequestToken(accessor);
      // Build the token pair to return
      result = new TwitterTokenPair();
      result.setToken(accessor.requestToken);
      result.setTokenSecret(accessor.tokenSecret);
   } catch (Throwable t) {
      throw new TwitterOAuthException(t);
   }   return result;}

4. Twitter will generate a Request Token and Token Secret and send them back
5. Save the Request Token and Token Secret off for later use.  We will use them again after the user has granted us access.
6. Build up a the URL for a user to access, sending them to Twitter, to grant us access.  This will be done by sending them to http://twitter.com/oauth/authorize?oauth_token=.  You can optionally add a callback URL on the parameters or just rely on the one that you entered on the Twitter site when registering your application.

public String getAuthorizeUrl(String token, String callbackUrl, Map callbackParameters) {
   Map parameters = new HashMap();
   parameters.put("oauth_token", token);
   if (null != callbackUrl) {
      parameters.put("oauth_callback", callbackUrl + "?" + asQueryString(callbackParameters));
   }
   return authorizeUrl + "?" + asQueryString(parameters);
}


7. If you user grants you access, then your callback URL will be called.
8. Upon recieving the callback, you now need to request and Access Token from Twitter.  This is very similar to step 3 except you will be using a different URL and the Request Token and Secret that you saved off in step 4.  You will sign your request with your consumer secret and the token secret and send the request for the Access Token to http://twitter.com/oauth/access_token.

public TwitterTokenPair getAccessToken(TwitterConsumerCredentials credentials, TwitterTokenPair requestTokenPair) throws TwitterOAuthException {
   TwitterTokenPair result = null;
   OAuthAccessor accessor = newAccessor(credentials);
   accessor.requestToken = requestTokenPair.getToken();
   accessor.tokenSecret = requestTokenPair.getTokenSecret();
   OAuthClient client = new OAuthClient(new HttpClient3());
   try {
      client.getAccessToken(accessor, HttpClient.GET, null);
      // Build the token pair to return
      result = new TwitterTokenPair();
      result.setToken(accessor.accessToken);
      result.setTokenSecret(accessor.tokenSecret);
   } catch (Throwable t) {
      throw new TwitterOAuthException(t);
   }
   result;
}

9. Twitter will generate an Access Token and Token Secret and send them back.  At this point, the Application is added to the users Connections tab.

10. Once you parse the Access Token and Token Secret out of the response, you can use them from that point forward to make Twitter API calls on behalf of the user that granted access.

That is it!  Now you have successfully acquired the Access tokens and can perform the Twitter REST API services for that user.  Because I have granted DigitalChalk the to update my status, the application can do so at its hearts content.

TwitterHttpCommand twitterCommand = new TwitterHttpCommand(credentials);
try {
   Map parameters = new HashMap();
   parameters.put("status", "Just finished OAuth integration implementation of Twitter and DigitalChalk in Java!  Woohoo!");
   response = service.execute(twitterCommand, HttpMethod.POST, accessTokens, "http://twitter.com/statuses/update.xml", parameters);
   printInputStream(response.getResponse());
} catch (TwitterOAuthException ex) {
   System.out.println(ex);
} catch (IOException ioex) {
   System.out.println(ioex);
}

UPDATE: I have been asked to include the code for this experiment.  You will need to download the Java OAuth Library and include it in the classpath as well as enter your Consumer Key and Secret into the twitter.properties file.  The Test.java class should walk you through how I used it to test access to Twitter.  Let me know what you think and if you would like to see anything else added.  You can download the code here: http://bit.ly/Pz55C

Playing with Eclipse and Amazon Web Services

It has almost been a week since Amazon released its AWS Eclipse plug-in, but today was the first time I had the chance to really play with it. While much of its functionality is available through the AWS Managment Console, it is really nice to have the Eclipse views right there in your development tool so that you don't have to leave it. This will enable me to monitor our servers more frequently. You almost tend to forget how much you have running in the cloud sometimes! I really wish that the plug-in would have included the ability to link multiple AWS accounts to a single Eclipse instance as I have several accounts that I need to monitor. Installing the plugin was very easy, but there was a gotcha that took me a couple of minutes to figure out. Once I had everything installed and restarted Eclipse, I immediately went to the EC2 Instances tab. I tried to right click on one of the instances and open a shell to the instance. It did not work. The red X's that you see beside the key pairs in the image below should have clued me in before I tried.

It was missing the private key pair file and therefore could not launch the shell. There was no menu item on this view that would allow me to add the key pair file and the general setup has no indication of being able to add them. To correct the problem, I had to go to the EC2 AMIs tab and attempt to launch an AMI. Once you do this, you are presented with a launch screen and a listing of all of your key pairs. If you have existing key pairs, you can right click on one of them, as shown in the screenshot below, and choose "Select Private Key File..."

This allowed me to choose my private key file and the green check marks appeared in the EC2 Instances view. This allowed me to open a shell right from within that Eclipse view. After that, I was off to the races and everything worked exactly as stated on the AWS Eclipse page.

Bruce Kerr - Tech's Weird Al

This is a great way to start the day. A simple little ditty from Bruce Kerr at Sun Microsystems on Cloud Computing. I hope you all enjoy.

AppEngine to support Java

While I enjoyed programming in Python to complete the Juicy Ideas Website on AppEngine, I am really looking forward to trying Java out on Google's cloud computing platform. It was one of the most requested languages to be supported in the AppEngine forums. It will be really nice to test the real power of AppEngine with one of my primary languages. It is just much easier when you know all of ins and out and details of the beast you are trying to tame. Java on AppEngine should be announced officially very soon, even before the May Google I/O event. One of the things that I will be most interested in seeing is what libraries will be available to the developer. The available Python are very restricted and controlled and I am hoping that we do not see the same for Java, but I am guessing we will. There are so many utility jar files, such as Xalan and Xerces, all the Apache Commons libraries, and scores of projects on SourceForge. Will GAE allow me to deploy a WAR file with everything that I need in it? That would definitely be nice. The other fun topic will be the mapping to BigTable. I am looking foward to seeing how that is going to work. The excitement builds.... what do you hope to see?

My First SXSW

I just got back from my first South by Southwest conference and I have to say that it certainly lived up to expectations. I have been hearing about this conference for a couple of years now and how I must go, but this is the first time that I have actually made it. If you are a young start-up web company, this is the show to be attending. There were some great booths and lots of people to talk with and share ideas. I did attend several panels and talks and I would have to say that my favorite had to be Steve Souders talking about his new book "Even Faster Web Sites" and all of the tips and tricks you can do with loading Javascript, CSS, Images etc. It was very useful and included some great details that you really gloss over (but really shouldn't) when you are trying to get a product out the door. I am pre-ordering the book at Amazon.com so that I can make sure I apply his principles. I also went to a panel on Version Control because I wanted to dig in deep with CVS and SVN and figure out why everyone is so hot on SVN. It was fairly disappointing because it was extremely high level and not much meat to the discussion. I was interested in hearing what Matt Mullenweg, of WordPress.com, had to say since he was on the panel. He was by far the most knowledgable on the panel and it showed. He actually did a demonstration of an SVN checkin that propogated to all of the WordPress.com servers and then he did another checkin to fix the change. It was the first time I had seen someone try to do a live commit to many live production servers during a presentation to 200 people. Even though it was a trivial change, it was still cool to see someone that confident in the systems they have in place. The last panel that I went to for the week was the Cloud Computing panel. It was particuarly interesting because you had Amazon.com, Google and Microsoft sharing the stage talking about their different platforms. It was obvious that Amazon had the most complete architecture right now, but I am looking forward to seeing what Azure comes out with in the near future. One of the highlights of the conference was being able to sit down one on one with Werner Vogels, the CTO of Amazon.com, and talk about how we are using the services at DigitalChalk and give him some feedback on our experiences. There were many other experiences that will always stand out, like meeting Guy Kawasaki and Robert Scoble and of course all of the parties. Needless to say, I will be going back next year.

GrandCentral == Google Voice

A number of people have been asking me how to get into GrandCentral and I must say that it wasn't easy. I did sign up early on and asked to be notified when it became publically available. I was lucky enough to run into a person that had the chance to invite a single person into the private beta. It has been a great service so far and I am happy to say that it looks like it will soon be available to everyone as Google Voice. The site says that current GrandCentral users will be asked to upgrade their accounts to Google Voice, but unfortunately I have not been able to do that yet on the site as I keep getting an "Invalid Request" message when I attempt to log in. Om Malik, a user of GrandCentral that is already made it into Google Voice, reports that the following features are available:

1. You can use your GrandCentral number to send and receive SMS messages, and have them forwarded to your current wireless phone. You can send messages from the mobile or from the phone.
2. Make phone calls using the web or your mobile/landline phone.
3. You can get transcripts for voicemails left on Google Voice. These transcripts, based on internal Google technology currently being used by GOOG-411 service, can be sent to you via SMS.
4. Create conference calls by dragging phone numbers onto existing calls. This will be useful for small businesses and web workers.
5. Free calls to all U.S. numbers. You can make international calls but that will cost you, depending on the country you are dialing.
   

Looks like it will be a great solution for the "cloud worker" that is always on the go and looking to streamline everything. I am looking forward to seeing the new interface.

Trip to Google

We had an absolute blast on the trip to the Googleplex in Mountain View, CA. Last week I had the pleasure of traveling out to Google's headquarters with the winning team of the Juicy Ideas Competition.  Ryan Klinger, Spencer Price, Andrew Drake and Justin Henry from Appalachain State University won the competition in the Western North Carolina region and at the national level.  The team was able to take a tour of Google headquarters, get shuffled around in a limo, and present their winning idea at a Google TechTalk.  The weekend was absolutely amazing and lots of thank yous and props go to Mary Radomile, Susie Vaks and Matt Dunne of Google, Pam Lewis of AdvantageWest and Dale Carroll, now with the State of North Carolina as Deputy Secretary.
I am happy to say that the we also had a couple of meetings that were focused on the development of next years competition.  I am excited that we have started the process again and I know it will be just as fun next year.  Get ready, because it is going to be bigger and include many more schools nationwide.  So, if you think you have the next Juicy Idea that will change the way that we live and do business, watch for the announcement of the competition at your local college campus.
Because DigitalChalk was working on the technology behind the competition and putting that on Google's AppEngine platform, it was especially interesting to get to speak with Pete Koomen, one of two product managers working on the platform.  We were able to discuss what the experience was with the platform this year and what will make it even better next year.  At one point, during the registration for the competition, we were experiencing 25 hits/second on the AppEngine site for the competition and the platform handled it with grace.  Pete was able to share some of the experiences that they have had in the past year on the platform and some of what is coming in the future.  We had a great discussion on the future of cloud computing and what might be some ways to help make a common platform across the different cloud providers.  I will be looking at implementing some of what we talked about and releasing it to the open source community if I am successful.  Needless to say, I am very excited about the competition next year and I can't wait to see what new ideas we see!

Rapidly Processing Video in the Cloud

Cloud computing is more than just a buzzword to those of us that rely on it everyday. It really makes computationally expensive tasks fun to tackle and much easier to pull off.  We had a very successful weekend processing a lot of video, and I thought I could take this opportunity to give an example of how the cloud computing model really saves in the end.  To give you a little bit of background, we needed to rapidly process some of the videos uploaded by our users this weekend in such a way that they would not even "feel" the difference.  We put Amazon Web Services to work on the task and accomplished what would have taken an estimated at 286 computing hours in 10.  Using a little SimpleDB, S3 and EC2 magic, we threw the problem to a grid of 20 EC2 instances that segmented and ripped through the videos in a fraction of the time it would have taken us otherwise.  What is so special about that?..  For us it was a great time saver and money saver.  I have included a screenshot of the AWS Management Console when we started the processing and after it was done and the instances were terminating.

If you have jobs that would take more than a couple of hours to do, it would be well worth your time to take a dive into the Amazon Web Services model.  The Management console made it very easy to start all 20 instances with a single click and take them down afterwards.  We used the new ability to copy content in S3 to a working bucket and the data storage of SimpleDB to help keep our EC2 instances from double processing.
I get a lot of requests to help people understand and harness the power of the cloud, and there is so much that you can use it for that the use cases are varied for every customer.  While our production systems are built to do live scaling and processing based upon load, I thought that this stand alone problem would be one of the more typical use cases to share.  It really shows that an isolated problem can be solved much faster with more resources very cheaply.

TIGR Team looks at Cloud Computing

As part of the transition to President Obama, a Technology Innovation and Governmental Reform Team (TIGR Team) has been setup. This would be a great team to be on and to work with. Obama effectively used the internet to win the election and it now appears that he has put together a team to continue to use the technology that is available now to help bring our government up to date. That is a huge undertaking, but it will be fun to watch and experience. I think that every state should have their own TIGR team to help reform their state. I have personally been talking with every person I can grab in the local, state and national political arena to drive this point with them. We can use cloud computing and web services to provide all the necessary computing infrastructure and data to anyone that needs to use it. Take a look at this video for a small glimpse into what is being talked about.

Skype BETA for Mac is out

Skype has resently posted a BETA (2.8.0.324) download of their client on their site, but I could not find a link to navigate to it on their main site. A little digging around and trying different URL combinations and I scored. Try http://www.skype.com/download/skype/macosx/beta if you want to grab the BETA now. The real meat in this release is that it now includes screen sharing and sports a slightly new interface for the video window. The picture below shows the new interface and a screen shot of our website while chatting with someone else.

AWS Management Console

Amazon announced today the beta release of its management console.  The use of the console, which can be found at https://console.aws.amazon.com is free and very easy to use.  If you are familiar with the services offered by Amazon Web Services, you will be up and running in no time on their management console.  The console currently allows you to manage your EC2 instances, Elastic IPs, Security Groups, Key Pairs and Elastic Block Stores.  Launching a new instance and even viewing the console output are only a click away.  This will be extremely helpful in just getting a nice consise high level view of the systems we are running in the Amazon cloud.