On The Cloud : Stay On Top of News Around Cloud Computing

We have created a custom News Paper called “On The Cloud” to stay on top of News around Cloud Computing and platforms of interest to us. As of now, we have included news on the following topics :

• Google App Engine
• Microsoft’s Windows Azure
• Amazon Elastic Cloud
• Force.com and
• all general topics around Cloud Computing

Here is a screenshot of the news paper on NewsCred.com

NewsCred allows anybody to create their own news paper based on topics of interest. Unlike others, NewsCred also allows to write your own editorials that will go with the news paper.

Software Developer Ad: Any clue who are they looking for?

Its very funny to look at job advertisements for Software Developers, particularly in the enterprise. Many times those requirements look like are actually written by completely non-technical business managers, who have no clue about what they really need.

And its quite shocking that many IT consultants garnish their resumes to fit to that profile. They get short listed, interviewed and selected to work on those projects. Something, very hard to explain, but is a norm.

See a sample requirement I just received:

The requirement says they need an experienced IT Professional with 10 years of solid IT development experience with the following skill set:

"Development Life Cycle, C++, Oracle (SQL & PL/SQL), Java (JDK 1.4/1.5), J2EE , Servlets, JSP, JDeveloper, Client/Server, Intranet, Internet, Multi tier enterprise applications, CVS, UNIX, Oracle Application Server, Twiki, Windows & Unix OS, JDBC, HTML, UML, XML, XSLT, HTML, XMLSpy 3.5 Tools, Schema, HTTP & JavaScript.

Any clue, who are they looking for?

Grady Booch : Why Engineering?

These are some of the most wonderful words ever said about the profession of Software Development. (I made some text in bold just to highlight)

Software is invisible to most of the world. Although individuals, organizations, and nations rely on a multitude of software-intensive systems every day, most software lives in the interstitial spaces of society, hidden from view except insofar as it does something tangible or useful.

Despite its transparency, as Bjarne Stroustrup has observed, “our civilization runs on software.” It is therefore a tremendous privilege as well as a deep responsibility to be a software developer. It is a privilege because what we do collectively as an industry has changed and will continue to change the world. It is a responsibility because the world in turn relies on the products of our labor in so many ways. In the context of that labor, software is perhaps the ultimate building material: it springs from pure thought and is intrinsically malleable, yet it can be made manifest in our hardware systems, limited only by our vision (and certain immutable laws of physics and software). As software professionals, we seek to develop and deploy useful systems of quality in a manner that reduces the distance from vision to execution. That the fruits of our labor are transparent to the world is as it should be: users want results and value, not more technology. For this reason, the primary challenge of every software development team is to engineer the illusion of simplicity in the face of essential complexity.

Handbook of Software Architecture – Grady Booch

The following Video by Grady Booch & co discusses the above quote while explaining the essence of software development in the context of Engineering.

Isn’t it fair to say the characteristic of profound understanding is the expression of simplicity and elegance. Well, Grady Booch exemplifies that when it comes to Software development.

Buzz In The Cloud : Cloud Computing Is The New Web2.0

These days, not a single day goes by without talking, hearing or reading something about Cloud Computing. It appears, the whole cloud (I mean the world; don’t blame me for overusing the word ‘cloud’, I just want to make a point) is totally excited about ‘Cloud’ suddenly.

In a way, Cloud Computing is the new web2.0, in terms of hype, buzz and activity all around. Every business has something to offer ‘on cloud’. Every consultant is busy selling strategies to cash in on the cloud. And every analyst is preparing a report on the cloud. Every developer is trying to equip to engineer applications on the cloud.

Simply put ..

What is really Cloud computing is all about? Why not look up the definition of Cloud Computing by going to the Encyclopedia on the cloud (??), Wikipedia.

The term cloud is used as a metaphor for the Internet, based on how the Internet is depicted in computer network diagrams and is an abstraction of the underlying infrastructure it conceals. – Wikipedia

Cloud computing is an example of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet. – Wikipedia

Though Cloud Computing could be everything for everybody (just like Web2.0), most stakeholders would agree to confine Cloud Computing in three forms.

Three Forms of Cloud Computing

1. Software-As-A-Service (SaaS)

Refers to Software Applications offered over the internet as a service. SalesForce.com is an excellent example and one of the most successful SaaS offerings. BaseCamp for Project Management, Google Apps are few other successful and well known SaaS offerings.

2. Infrastructure As A Service (IaaS)

Refers to Servers, Storage and Networks offered as a service over the internet.  Most Internet hosting providers fall in to this space. Amazon ECS, Rackspace, AT&T, Akamai are very good examples. They offer servers, databases, storage area networks, networking gear, content delivery networks as a service.

3. Platform As A Service (PaaS)

These providers offer Infrastructure as a Service with a limitation or leverage (depends on what you are looking for) that all applications must be built and run on top of their often proprietary platforms or technology stacks. PaaS inherently include IaaS. Google App Engine, Microsoft Azure, SalesForce Force.com are fine examples for this service.

SaaS and IaaS have been offered in the market for a while. Though there are lots of advances in the technicalities of managing them and lot more new players in the market, these services are not entirely NEW.

What is new in Cloud Computing is the third service, Platform-As-A-Service (PaaS). Not only Infrastructure to run your Business applications but a full platform to develop those applications. It is quite unlikely that these PaaS offerings (at least initially) will not provide a smoother migration for Enterprises to adopt Cloud Computing as this means re-developing most of their business applications.

Easier Adoption Path for Enterprises

For most enterprises, the only option that make sense at this point is IaaS. Instead of running their own data centers, applications can be moved to a public Cloud or a private cloud or managed cloud with little to no impacts. Cost of migrating to a Cloud will be quite insignificant and offer the best level of cost savings, while not forcing a vendor lock-in as applications run on generic platforms. This is the best option.

Or Enterprises can migrate to SaaS offerings, if there are comparable applications are available. For instance, instead of running your own CRM solution on premises, can migrate to SalesForce.com.  But it is a big question for how much generic application like SalesForce.com can offer unique capabilities for each organization, a primary reason why Enterprises developed their own software in the first place. Vendor + Application lock-in is implicit. All Employees and Customers need to be retrained in new sets of applications. Extensive customization is required.

The latest entry in to Cloud Computing, the PaaS is the most Unlikely path of migration of any Enterprise application to the cloud, as this needs re-architecture of their applications and Migration of application software as well  along with data migration. And this would be quite expensive and time consuming for any non-trivial, uncommon business application. Vendor + Platform lock-in is implicit.  Need to retrain Software Development groups and it could be quite extensive as the Platform could be significantly different from generic platform most software developers are comfortable with.

You can read a little more on the Wikipedia. Or for that matter anywhere on the Cloud !!!

Cloud For Enterprises : Why They Must Consider Cloud?

Some analysts argue that 'Cloud Computing' is only for small businesses, ISVs and web startups where there is a need to start low (and cheap) and scale later. The underlying assumption behind the argument is that, in enterprises (read as large businesses with a variety of platforms and applications), demand is quite predictable and doesn't change so often.

Well, that is not quite true. While number of Employees in the organization may not change so often, their usage patterns are much like any web application startup.

For instance, take an enterprise with 25,000 employees. The enterprise IT department has developed an intranet web application for Performance Reviews. The web application will be accessed by Employees and Managers at the beginning of the year to set Goals. And will be accessed in January/February to appraise performance with respect to the set goals. The application will also be accessed sparingly by employees and managers through out the year to make necessary status updates, follow ups on action plans etc.

Typically, IT department will design the system such that it can handle the load during the worst peak hours.  In most Enterprises, the most productive day to get any thing done is the last day. So, the system shall be designed to handle the load of all 25,000 employees accessing the system in a single day.  Assume that need 5 servers to handle the load. So what happens to those 5 servers for the rest of the year?

If the enterprise uses a cloud service (could be an in-house, partner or a public cloud service), the firm can run the least possible configuration throughout the year and scale up to 5 servers during the last few days of the year or whenever they need (like a mid-year reviews in June/July).

There are many Enterprise applications that are normally run with a fixed infrastructure designed to take care of peak loads. For instance Expense reporting applications (normally accessed only at end of month), Payroll applications(normally accessed only at the beginning of the month or once in a fortnight) and even Email servers(normally accessed only during office hours). Most Enterprise applications are not accessed all-the-time. Cloud service is a perfect target for all those applications and could save an enterprise a huge chunk of their budgets.

So, Cloud Computing will benefit businesses large and small alike. In many cases, larger the enterprise, larger will be the savings if Cloud computing is used.

Scaling Rails Applications on Amazon Cloud

The application demonstrated in this talk is hardly handling about 500 users, but demonstrate some interesting points about scalability that could be almost taken for granted in Cloud hosting services like Amazon. Also a good introduction to Cloud if you are not already on cloud 9.

Scaling Rails Applications In The Cloud

Listen to the MP3 of the presentation here.

Watch slide 23, the non-RDBMS bonus carefully. If you have read about my previous post on this topic, Shifting Away From Tables, To Entity Models, the point comes home pretty convincingly in this presentation.

Google App Engine : Lazy Data Migration With Versions

If you have already used Google AppEngine to develop an application, you would have already scratched your head around data migration. And Google App Engine is not that great when it comes to Agile style iterations of web development. In its own way, it forces you to design models up front instead of making life easier evolving over time.

I have been developing a community application, ‘Yes to Politics’ for friends in Andhra Pradesh, India to interact with politics in some strangely different way. More about this app later, let me share about data migration.

I have around 28000 entities in a model and trust me, I tried to cover all my needs in the design of the model up front.  Well, software development doesn’t work that way. I realized I needed another property in the model and I needed to give some default value too.

When you add a new property to an existing model, Google App engine doesn’t fill the default value for the existing set of entities. So when you access that new property on existing rows, you will meet with exceptions as they never existed for them. You can check if that property exist before accessing it. Well, Python provide no way to do it. The only way to find whether that property exists or not is by accessing it and catching an exception when it doesn’t exist. Not a cool way, but that’s almost the best method we got.

And, you can not use that new property in queries too that need to search in the existing set of entities. There is a way we can fix it.

Define your new property and then loop through all your entities and set this new default value.  This is no easy job to do on Google App engine. First, you need to setup a new URL and a HTTP handler to take care of this maintenance. Second, you can only update so many entities in one go without exceeding restrictions in terms of time and processing power. So you need to split the task in to easier bits, say, update 10 entities at a time and create a handler to auto refresh every few seconds to take care of all updates. And then run that handler in the browser and wait till it is finished.

I have about 28000 entities and that would mean I have to call that handler almost 2800 times (10 entities at once) and better give some wait between calls to make App Engine restrictions happy. For my model, 5 sec between each call worked fine. Any quicker, App Engine throws an exception. And it took about 25 minutes for me to finish the process.

I thought it will be anyway one time task so didn’t regret waiting that long. But then after all that is done and happily using it for a couple of days,  I found that I had to add yet another property.

This time, instead of doing that hard way, I have decided to do something different. Instead of adding the new property I was thinking of, I added a simple integer that will act as a version number for the entity. I followed the same as above and waited another half hour to get it updated. And for the actual property that I wanted to add, I added the new property but deferred updating the property with a default value until that entity is being used. The lazy way. So I just need to update my queries with this new version number logic but don’t have to really update all my data at once.

I realized later it was an excellent move. As not all my entities really need that new property added. Whichever entity needs it, will get it when it is accessed for the first time. Your data migration now becomes highly scalable. Now all my models, I begin with a version number, so that I never have to worry so much about data migration when I decide to make changes to my models.

This is not without a downside. We have an additional property in the model and a little overhead of a version number comparison every time entity is accessed. If you are continuously adding properties that may not be required for all entities, then the storage space you save could easily outweigh this new additional property.

But, you decide whether the flexibility of scalable data migration it provides is worth the weight and hassle.

Expanding the Cloud: The Amazon RDS

Excellent notes on various Data base systems available on Amazon Cloud and explanation of key constraints that must be kept in mind while deciding on a system for a given application. Note that there is no one system that fits the best for all applications. Architects must pick the choice that is right in a given context.

Excerpts from the article that highlight various options available : Amazon S3 (Simple Storage Service), Amazon SimpleDB and Amazon RDS

Early on already the distinction was made between key-values storage systems and structured data management. Key-Value storage systems play a very important role in the Amazon architecture and this has ultimately led to the creation of the Amazon Simple Storage Service (Amazon S3). Amazon S3 addresses the need for a highly scalable and reliable Key-Value data storage system while shielding customers from all the complexities such as geo-replication, capacity planning, and performance management at high scale.

Structured data management systems are traditionally served by relational databases but these sophisticated systems have their limitations, especially when it comes to scale and reliability. Often they also require tremendous expertise to operate efficiently and reliably especially when scaling up. Of course, a significant portion of the structured data world does not require RDBMS features such as complex transactions and relations, and can be served by a simpler, much more agile system. Such a simple structured storage system for example does not require the use of a rigid schema and can allow attributes and indexes to be adapted on the fly. This system has led to the creation of Amazon SimpleDB where its customers get the benefits of such a simple scalable structured storage system without having to worry about replication, backups, buffer cache optimizations, databases resizing, etc

There are a several applications and services that do need the feature richness of an RDBMS. Until now they were served through the use of the Relational Database AMIs that are available for Amazon EC2. These AMIs can be launched to create a compute instance with database technologies such as Vertica, Oracle, DB2, SQL Server, Sybase, and PostgreSQL. These RDBMS are best used in concert with the Amazon Elastic Block Store (EBS) to create a scalable and reliable storage volume that can be used for persisting the databases.

…

For these customers who require a relational database but do not have a need to exert complete administrative control over their database server, there is now another option: the Amazon Relational Database Service (Amazon RDS)."

Source : Expanding the Cloud: The Amazon Relational Database Service (RDS)

Street Views Can Make Google (India) Maps Outstanding

Google India recently launched Google Maps with navigation based on landmarks, with landmark information fed by the local community in respective cities. Its quite amazing to see how crowd sourcing could be leveraged where structural data is hard to retrieve.

There is no doubt, that this kind of landmark based directions first introduced in India by Routeguru almost a couple of years back. But, we are quite excited to see Google doing this as we could visualize so many other possibilities that could redefine maps for navigating Indian roads.

Street Views

For instance, what a huge difference it will make if Google could add "Street Views" to those directions? Landmarks in text is something. But a picture of that landmark really makes it the most intuitive way of looking at the map.

Instant offers/coupons en-route

Since directions are based on landmarks, that include local businesses, instant offers and coupons from those businesses en-route would be quite beneficial to all.

So here is our request to Google India Maps team : Please  add “Street Views” to make Google India Maps with Land Marks totally outstanding and intuitive to local communities.