Engineering of Failure

This article is for all the engineers who design and develop products across the world for the CONSUMERS. True engineers have this urge, a strong desire to keep developing new stuff, which not only makes this world more beautiful but also helps them pay their bills. Little we are aware that this “engineering” is designed to FAIL. The products we engineers develop or the technology we use has a “Life Span” or an “Expiry Date” which is dictated by the Consumer Society and Corporates.

I am a movieaholic! Of all the things in the world, there is one thing that I can do at least once a day is watching movies. Speaking of movies, if you’ve never watched FIGHT CLUB please do it. It teaches us the basic mistakes that we do in our everyday lives.


Now, let’s discuss a simple everyday problem which we all face : Strong desire to buy a new smart phone.

You’ve got the latest and costliest phone of the market, with all the shiny features but within a year you will plan to buy another one. Trust me, there is nothing wrong with your mind, this is how our brains are dictated (by the society) to work. This “strong desire” is cultivated or rather implanted in our brains by the business minds of our world. It is like the idea of fashion and music. The fashion to sport a particular style of clothes along with listening to a particular style of music has been the default identity of a period of time or an era. For example Bell Bottom Pants and Rock Music was THE thing in the 70s – 80s. Now is the time of ever changing and fast evolving technologies, smart phones, gadgets, social media and Electronic Dance Music. While change is good, we need to understand why it is good? and How it is practically achieved? Well, we achieve this by a combination of FAILURE of something and marketing of something else as BETTER.


All the above things may sound obvious but they are NOT. There is a very interesting theory behind this called as The Light Bulb Conspiracy . Following is a famous documentary related to this theory:

According to this theory, products are designed to fail after certain period of time. Same is true with technology. No matter how talented are the engineers of any era, their talent will always be dictated by the consumer based society and business men who think of nothing but profit. So whenever something great is created it will be degraded somewhat before presenting it to the world, so that the product dies slowly, making way for something new which could lead to monetary profit rather than a technical one.

A classic example is of Apple Inc. They tell the world that there is a sheer brilliance in their product but after a while they will mark the same product as ‘inferior’ and will stop supporting them, forcing their own customers (rather consumers) to buy their new, costlier products. The aura and hype of Apple Inc. is another strategic plan to keep the company’s sales up.

Similar trends can be seen with the technology. Take for example Web Development. Every year something new comes into the picture for developing “cutting edge” websites. But popular frameworks or technologies of today may vanish tomorrow, engineers will never know. This is the direct result of the “Light Bulb Conspiracy”. Because of these “profit making” strategies, Software Engineers are in constant fear of losing their jobs leading them to keep learning new technologies every now and then. The consequences paints a picture with it’s own pros and cons. On one hand there is always restlessness in the IT world but on the other hand it keeps and average IT professional’s brain alive and active.

One argument may be that new technologies provides better and efficient Software Development techniques. This is partially true. While the new programming languages / techniques / tools / frameworks do provide easy approaches of Software Development, this may also kill the jobs of the skilled IT professionals who have dealt with the ‘hard way’ of Software Development.


The IT industry and technical companies are increasingly becoming clever in selling “old stuff in new packaging”. The biggest example is Cloud Computing. As an official article by IBM says and I quote :

It (cloud computing) was a gradual evolution that started in the 1950s with mainframe computing.

Multiple users were capable of accessing a central computer through dumb terminals, whose only function was to provide access to the mainframe. Because of the costs to buy and maintain mainframe computers, it was not practical for an organization to buy and maintain one for every employee. Nor did the typical user need the large (at the time) storage capacity and processing power that a mainframe provided. Providing shared access to a single resource was the solution that made economical sense for this sophisticated piece of technology.

This is one big HOPE that IT has given us that even if according to the Light Bulb Conspiracy a particular technology is meant to die after sometime, we can always redesign the business model and present it with some other context to the world to consume it again. The people of the world will happily consume it if an urgent necessity and “fashion to consume it” is shown to them in a proper manner. Then we too can sit back with a popcorn and enjoy the show!



Python | A simple prototype for tracking human movement

At times social networking sites, job portals and even the government need to keep a tab over the movement of the people from one place to another (or one nation to another). Governments do this analysis on a regular basis to check the rate of urbanization.

Assumptions: Let’s take the example of a Social Networking site. Here I have assumed that we have the following table (In Hadoop or any other Big Data framework of your choice) having the data of people moved from one place to another.

id                                           int #Autogenerated ID
uid                                         int #User ID
some_other_id                  int
cur_location                       int  #Location ID mapped to “Location”  table

So everytime an user updates his/her location we do an INSERT into the database rather than an UPDATE. This way we have the record of the user’s movement in chronological order which can be retrieved using his “User ID”.


Retrieve data [Hadoop Layer using Hive]

Process it into meaningful data [Python]

Store results [MySQL] (So that it is easy to use for displaying purposes)

Python Mining Engine:

Here is a simple python program I wrote for this simple application. In actual real world scenarios you might have to use or invent some complex data mining algorithms. For the purposes of this blog post I successfully used the python script for processing 35,000 records successfully. Here is the complete code (you can also check it out here ):

__author__ = 'Abhay Gupta'
__version__ = 0.1

import pyhs2
import time
import MySQLdb
import datetime
import sys
import pytz

def DATE_TIME():

    return datetime.datetime.strptime(str(DATE_TIME()).split('.')[0], '%Y-%m-%d %H:%M:%S')

def DrawSpinner(counter):
    if counter % 4 == 0:
    elif counter % 4 == 1:
    elif counter % 4 == 2:
    elif counter % 4 == 3:

def neighourhood(iterable):
    iterator = iter(iterable)
    prev = None
    item =  # throws StopIteration if empty.
    for next in iterator:
        yield (prev,item,next)
        prev = item
        item = next
    yield (prev,item,None)

#hive cursor
def get_cursor():
        conn = pyhs2.connect(host='',
        return conn.cursor()

def get_mysql_cursor():
        conn = MySQLdb.connect(user='db', passwd='',
        return conn.cursor()

def get_records():
        cur = get_cursor()
        cur.execute("select * from user_location_history")
        #Fetch table results
        return cur.fetchall()

def get_user_movement():
        location_dict = {}
        #Fetching all the records
        records = get_records()
        counter = 0
        for record in records:
                counter = counter + 1
                if location_dict.has_key(record[1]):
                        location_dict[record[1]] = [int(record[3])]
        return location_dict

#For performance improvements use list instead of dictionary as we don't need count of the users here
def prepare_movement_data():
        city_movement_data_dict = {}
        user_location_dict = get_user_movement()
counter = 0
        for user_id, user_movement_path in user_location_dict.iteritems():
                counter = counter + 1
                if len(set(user_movement_path)) > 1:
                        if city_movement_data_dict.has_key(tuple(user_movement_path)):
                                city_movement_data_dict[tuple(user_movement_path)] = city_movement_data_dict[tuple(user_movement_path)] + 1
                                city_movement_data_dict[tuple(user_movement_path)] = 1
        return city_movement_data_dict

def store_mining_results(unique_movement_map_tuple):
        sql_query = None
        insert_flag = False
        update_flag = False
        cur = get_mysql_cursor()
        for prev, current, next in neighourhood(unique_movement_map_tuple):
                #Execute query
                cur.execute('select * from user_flow_location')
                #Handling the empty table case
                fetched_data = cur.fetchall()
                if len(fetched_data) == 0 and prev is not None and current is not None and prev != current and current != next:
                        sql_query = 'insert into user_flow_location(location_from, location_to, count)\
                                                values('+  str(prev) + ',' + str(current) + ', 1 )'
                        insert_flag = False
                        update_flag = False
                        for record in fetched_data:
                                if record[2] == prev and record[3] == current:
                                        update_flag = True
                                        sql_query = 'update user_flow_location set count = ' + str(record[3] + 1) +\
                                        ' where id = ' + str(record[0])
                                elif prev is not None and current is not None and prev != current and current != next:
                                        insert_flag = True
                if update_flag == False and insert_flag == True:
                        #Insert only if the entry doesn't exists
                        #A quick fix. It can be improved later.
                        cur2 = get_mysql_cursor()
                        cur2.execute('select * from user_flow_location where location_from = ' +\
                                         str(prev) + ' and location_to = ' + str(current))
                        if len(cur2.fetchall()) == 0:
                                sql_query = 'insert into user_flow_location\
                                                (location_from, location_to, count)\
                                                values('+  str(prev) + ',' + str(current)+', 1)'

if __name__ == '__main__':
        start_time = time.time()
        #spinner = spinning_cursor()
        print 'Preparing data...'
        city_movement_data_dict = prepare_movement_data()
        print '\nData prepared for mining.'
        print 'Processing data and storing results...'
        counter = 0
        for unique_movement_map, unique_user_count in city_movement_data_dict.iteritems():
                #print str(unique_movement_map), ' : ', str(unique_user_count), ' user(s) relocated through this path'
                counter = counter + 1
        end_time = time.time()
        print '\nData mining complete!'
        print '\nTotal execution time = ', str(end_time - start_time), ' seconds\n'

How to import all the files from a directory using Python





STEP 1 : Execute the following command in the folder from which you need to import files from:

$ touch

Or simply create a blank file in the folder.

STEP2 : Prepare your The idea is to enable __init__ to know about all the files of your directory.

Paste the following code in

import os
modules = []
file_list = os.listdir(os.path.dirname(__file__))
for files in file_list:
    mod_name, file_ext = os.path.splitext(os.path.split(files)[-1])
    if file_ext.lower() == ‘.py’:
        if mod_name != ‘__init__’:

__all__ = modules

STEP 3:  Import in the desired file present on some other location

Include the following code on top your file:

import os
dir_name = ‘’
file_list = os.listdir(dir_name)
for files in file_list:
    mod_name, file_ext = os.path.splitext(os.path.split(files)[-1])
    if file_ext.lower() == ‘.py’ and  mod_name != ‘__init__’:
        exec “from {0} import {1}”.format(dir_name + files.split(“.”)[0], files.split(“.”)[0])