Engineering of Failure

This article is for all the engineers who design and develop products across the world for the CONSUMERS. True engineers have this urge, a strong desire to keep developing new stuff, which not only makes this world more beautiful but also helps them pay their bills. Little we are aware that this “engineering” is designed to FAIL. The products we engineers develop or the technology we use has a “Life Span” or an “Expiry Date” which is dictated by the Consumer Society and Corporates.

I am a movieaholic! Of all the things in the world, there is one thing that I can do at least once a day is watching movies. Speaking of movies, if you’ve never watched FIGHT CLUB please do it. It teaches us the basic mistakes that we do in our everyday lives.

fight-club-quotes-we-buy-things-we-dont-need

Now, let’s discuss a simple everyday problem which we all face : Strong desire to buy a new smart phone.

You’ve got the latest and costliest phone of the market, with all the shiny features but within a year you will plan to buy another one. Trust me, there is nothing wrong with your mind, this is how our brains are dictated (by the society) to work. This “strong desire” is cultivated or rather implanted in our brains by the business minds of our world. It is like the idea of fashion and music. The fashion to sport a particular style of clothes along with listening to a particular style of music has been the default identity of a period of time or an era. For example Bell Bottom Pants and Rock Music was THE thing in the 70s – 80s. Now is the time of ever changing and fast evolving technologies, smart phones, gadgets, social media and Electronic Dance Music. While change is good, we need to understand why it is good? and How it is practically achieved? Well, we achieve this by a combination of FAILURE of something and marketing of something else as BETTER.

aec2fa02573cb911b7ad6ea82e38df11

All the above things may sound obvious but they are NOT. There is a very interesting theory behind this called as The Light Bulb Conspiracy . Following is a famous documentary related to this theory:

According to this theory, products are designed to fail after certain period of time. Same is true with technology. No matter how talented are the engineers of any era, their talent will always be dictated by the consumer based society and business men who think of nothing but profit. So whenever something great is created it will be degraded somewhat before presenting it to the world, so that the product dies slowly, making way for something new which could lead to monetary profit rather than a technical one.

A classic example is of Apple Inc. They tell the world that there is a sheer brilliance in their product but after a while they will mark the same product as ‘inferior’ and will stop supporting them, forcing their own customers (rather consumers) to buy their new, costlier products. The aura and hype of Apple Inc. is another strategic plan to keep the company’s sales up.

Similar trends can be seen with the technology. Take for example Web Development. Every year something new comes into the picture for developing “cutting edge” websites. But popular frameworks or technologies of today may vanish tomorrow, engineers will never know. This is the direct result of the “Light Bulb Conspiracy”. Because of these “profit making” strategies, Software Engineers are in constant fear of losing their jobs leading them to keep learning new technologies every now and then. The consequences paints a picture with it’s own pros and cons. On one hand there is always restlessness in the IT world but on the other hand it keeps and average IT professional’s brain alive and active.

One argument may be that new technologies provides better and efficient Software Development techniques. This is partially true. While the new programming languages / techniques / tools / frameworks do provide easy approaches of Software Development, this may also kill the jobs of the skilled IT professionals who have dealt with the ‘hard way’ of Software Development.

awesome-fight-club-quotes-if-you-feel-like-shit

The IT industry and technical companies are increasingly becoming clever in selling “old stuff in new packaging”. The biggest example is Cloud Computing. As an official article by IBM says and I quote :

It (cloud computing) was a gradual evolution that started in the 1950s with mainframe computing.

Multiple users were capable of accessing a central computer through dumb terminals, whose only function was to provide access to the mainframe. Because of the costs to buy and maintain mainframe computers, it was not practical for an organization to buy and maintain one for every employee. Nor did the typical user need the large (at the time) storage capacity and processing power that a mainframe provided. Providing shared access to a single resource was the solution that made economical sense for this sophisticated piece of technology.

This is one big HOPE that IT has given us that even if according to the Light Bulb Conspiracy a particular technology is meant to die after sometime, we can always redesign the business model and present it with some other context to the world to consume it again. The people of the world will happily consume it if an urgent necessity and “fashion to consume it” is shown to them in a proper manner. Then we too can sit back with a popcorn and enjoy the show!

62454172

Python | A simple prototype for tracking human movement

At times social networking sites, job portals and even the government need to keep a tab over the movement of the people from one place to another (or one nation to another). Governments do this analysis on a regular basis to check the rate of urbanization.

Assumptions: Let’s take the example of a Social Networking site. Here I have assumed that we have the following table (In Hadoop or any other Big Data framework of your choice) having the data of people moved from one place to another.

id                                           int #Autogenerated ID
uid                                         int #User ID
some_other_id                  int
cur_location                       int  #Location ID mapped to “Location”  table

So everytime an user updates his/her location we do an INSERT into the database rather than an UPDATE. This way we have the record of the user’s movement in chronological order which can be retrieved using his “User ID”.

Architecture:

Retrieve data [Hadoop Layer using Hive]

Process it into meaningful data [Python]

Store results [MySQL] (So that it is easy to use for displaying purposes)

Python Mining Engine:

Here is a simple python program I wrote for this simple application. In actual real world scenarios you might have to use or invent some complex data mining algorithms. For the purposes of this blog post I successfully used the python script for processing 35,000 records successfully. Here is the complete code (you can also check it out here ):

__author__ = 'Abhay Gupta'
__version__ = 0.1

import pyhs2
import time
import MySQLdb
import datetime
import sys
import pytz

def DATE_TIME():
    return datetime.datetime.now(pytz.timezone('Asia/Calcutta'))

def FORMATTED_TIME():
    return datetime.datetime.strptime(str(DATE_TIME()).split('.')[0], '%Y-%m-%d %H:%M:%S')

#Spinner
def DrawSpinner(counter):
    if counter % 4 == 0:
        sys.stdout.write("/")
    elif counter % 4 == 1:
        sys.stdout.write("-")
    elif counter % 4 == 2:
        sys.stdout.write("\\")
    elif counter % 4 == 3:
        sys.stdout.write("|")
    sys.stdout.flush()
    sys.stdout.write('\b')

#Generator
def neighourhood(iterable):
    iterator = iter(iterable)
    prev = None
    item = iterator.next()  # throws StopIteration if empty.
    for next in iterator:
        yield (prev,item,next)
        prev = item
        item = next
    yield (prev,item,None)

#hive cursor
def get_cursor():
        conn = pyhs2.connect(host='',
               port=10000,
               authMechanism="PLAIN",
               user='hadoop',
               password='',
               database='test')
        return conn.cursor()

def get_mysql_cursor():
        conn = MySQLdb.connect(user='db', passwd='',
                              host='',
                              db='test')
        return conn.cursor()

def get_records():
        cur = get_cursor()
        cur.execute("select * from user_location_history")
        #Fetch table results
        return cur.fetchall()

def get_user_movement():
        #Initializing
        location_dict = {}
        #Fetching all the records
        records = get_records()
        counter = 0
        for record in records:
                counter = counter + 1
                DrawSpinner(counter)
                if location_dict.has_key(record[1]):
                        location_dict[record[1]].append(int(record[3]))
                else:
                        location_dict[record[1]] = [int(record[3])]
        return location_dict

#For performance improvements use list instead of dictionary as we don't need count of the users here
def prepare_movement_data():
        #Initializing
        city_movement_data_dict = {}
        user_location_dict = get_user_movement()
counter = 0
        for user_id, user_movement_path in user_location_dict.iteritems():
                counter = counter + 1
                DrawSpinner(counter)
                if len(set(user_movement_path)) > 1:
                        if city_movement_data_dict.has_key(tuple(user_movement_path)):
                                city_movement_data_dict[tuple(user_movement_path)] = city_movement_data_dict[tuple(user_movement_path)] + 1
                        else:
                                city_movement_data_dict[tuple(user_movement_path)] = 1
        return city_movement_data_dict

def store_mining_results(unique_movement_map_tuple):
        sql_query = None
        insert_flag = False
        update_flag = False
        cur = get_mysql_cursor()
        for prev, current, next in neighourhood(unique_movement_map_tuple):
                #Execute query
                cur.execute('select * from user_flow_location')
                #Handling the empty table case
                fetched_data = cur.fetchall()
                if len(fetched_data) == 0 and prev is not None and current is not None and prev != current and current != next:
                        sql_query = 'insert into user_flow_location(location_from, location_to, count)\
                                                values('+  str(prev) + ',' + str(current) + ', 1 )'
                        cur.execute(sql_query)
                else:
                        insert_flag = False
                        update_flag = False
                        for record in fetched_data:
                                if record[2] == prev and record[3] == current:
                                        update_flag = True
                                        sql_query = 'update user_flow_location set count = ' + str(record[3] + 1) +\
                                        ' where id = ' + str(record[0])
                                        cur.execute(sql_query)
                                elif prev is not None and current is not None and prev != current and current != next:
                                        insert_flag = True
                if update_flag == False and insert_flag == True:
                        #Insert only if the entry doesn't exists
                        #A quick fix. It can be improved later.
                        cur2 = get_mysql_cursor()
                        cur2.execute('select * from user_flow_location where location_from = ' +\
                                         str(prev) + ' and location_to = ' + str(current))
                        if len(cur2.fetchall()) == 0:
                                sql_query = 'insert into user_flow_location\
                                                (location_from, location_to, count)\
                                                values('+  str(prev) + ',' + str(current)+', 1)'
                                cur.execute(sql_query)
                        cur2.close()
        cur.close()

if __name__ == '__main__':
        start_time = time.time()
        #spinner = spinning_cursor()
        print 'Preparing data...'
        city_movement_data_dict = prepare_movement_data()
        sys.stdout.flush()
        sys.stdout.write('\b')
        print '\nData prepared for mining.'
        print 'Processing data and storing results...'
        counter = 0
        for unique_movement_map, unique_user_count in city_movement_data_dict.iteritems():
                #print str(unique_movement_map), ' : ', str(unique_user_count), ' user(s) relocated through this path'
                store_mining_results(unique_movement_map)
                counter = counter + 1
                DrawSpinner(counter)
        sys.stdout.flush()
        sys.stdout.write('\b')
        end_time = time.time()
        print '\nData mining complete!'
        print '\nTotal execution time = ', str(end_time - start_time), ' seconds\n'

Easy to make DIY 3D Hologram using Smartphone or Tablet | 3D Pyramid Hologram

Everyone loves Iron Man! And admit it, we all want Tony’s super cool lab, gadgets and of course Jarvis in our lives.

Sadly, most of us are not a billionaire like him and so cannot afford to own such things. But we can at least simulate (a bit) the super cool 3D holograms he used to design Iron Man suit etc.
Iron-ManAVENGERS

Star-Wars

Even in the cult classic Star Wars we have seen R2D2 projecting secret message in the form of a hologram, weird aliens attending meeting “virtually” etc etc.

How about creating one for yourself?
Here’s what you need :

  1. Old DVD case
  2. A paper cutter
  3. Graph paper or just plain paper
  4. Pencil and a ruler

IMG_0185

Steps to make your cool 3D hologram:

  1. Cut out a trapezium from a paper of the following dimensions:
    trapezium
  2. Place this shape on the CD case and cut out four such trapeziums. I suggest using a sharp cutting tool or knife. Use paper cutter for marking the outline. Be careful and do not break the plastic.
  3. Arrange these four plastic trapeziums into a pyramid and stick them together using adhesive tape or super glue.
    pyramid
  4. Voila! Your very own 3D hologram projecting pyramid is ready!

Now all you need to do is carefully place it over a smart phone’s or tablet’s screen while playing this video. You can try other videos too available online but this one is my personal favourite.

Check out my first 3D hologram pyramid too :

Courtesy