Big data, big analytics, big opportunity

[ani82v]

Analytics is to become the next big job generation sector. Not sure why it has not taken off in India yet.
When Indian IT companies are now registering avg growth of 10-15% instead of 30-40% couple of years back; not much focus is seen here. Or is it that Analytics hasn't taken off fully yet? Or is it because there not many statisticians in India?

Big data, big analytics, big opportunity | DAWN.COM


Data, data, every where
And not a byte to think


The world today is awash with data. Corporations, governments, and individuals are busy generating petabytes of data on culture, economy, environment, religion, and society. While data has become abundant and ubiquitous, data analysts needed to turn raw data into knowledge are in fact in short supply.

With big data comes big opportunity for the educated middle class in the developing world where an army of data scientists can be trained to support the offshoring of analytics from the western countries where such needs are unlikely to be filled from the locally available talent.

In a 2011 report, McKinsey Global Institute revealed that the United States alone faces a shortage of almost 200,000 data analysts. The American economy requires an additional 1.5 million managers proficient in decision-making based on insights gained from the analysis of large data sets. And even when Hal Varian, Google's famed chief economist, profoundly proclaimed that "the real sexy job in 2010s is to be a statistician," there were not many takers for the opportunity in the West where students pursuing degrees in statistics, engineering, and other empirical fields are small in number and are often visa students from abroad.

A recent report by Statistics Canada revealed that two-thirds of those who graduated with a PhD in engineering from a Canadian University in 2005 spoke neither English nor French as mother tongue. Similarly, four out of 10 PhD graduates in computers, mathematics, and physical sciences did not speak a western language as mother tongue. Also, more than 60 per cent of engineering graduates were visible minorities, suggesting that the supply chain of highly qualified professional talent in Canada, and to a large extent in North America, is already linked to the talent emigrating from China, Egypt, India, Iran, and Pakistan.

The abundance of data and the scarcity of analysts present a unique opportunity for developing countries, which have an abundant supply of highly numerate youth who could be trained and mobilised en masse to write a new chapter in offshoring. This would require a serious rethink for thought leaders in developing countries who have not taxed their imaginations beyond thinking of policies to create sweat shops where youth would undersell their skills and see their potential wilt away while creating undergarments for consumers in the west. The fate of the youth in developing countries need not be restricted to stitching underwear or making cold calls from offshored call-centres in order for them to be part of the global value chains. Instead, they can be trained as skilled number-crunchers who would add value to otherwise worthless data for businesses, big and small.

A multi-billion dollar industry

The past decade has witnessed a major change in the sectorial evolution of some very large manufacturing firms known in the past for mostly hardware engineering and now evolving into firms delivering services, such as business analytics. Take IBM for example, which specialised as a computer hardware company producing servers, desktop computers, laptops, and other supporting infrastructure. That was IBM's past. Today, IBM is focussed on analytics. It is spending hundreds of millions of dollars in advertising, trying to rebrand itself as a leader in business analytics. In fact, it has divested from several hardware initiatives, such as manufacturing laptops, and has instead spent billions in acquisitions to build its analytic credentials. For instance, IBM has acquired SPSS for over a billion dollars to capture the retail side of the Business analytics market. For large commercial ventures, IBM acquired Cognos to offer full service analytics.

In 2011 alone, the business analytics software market was worth over $30 billion. Oracle ($6.1bn), SAP ($4.6 bn), IBM ($4.4 bn), and Microsoft and SAS each with $3.3 bn in sales led the market. It is estimated that the sale of business analytics software alone will hit $50 billion by 2016. Dan Vesset of IDC, a company specialising in watching industry trends, aptly noted that business analytics had "crossed the chasm into the mainstream mass market" and the "demand for business analytics solutions is exposing the previously minor issue of the shortage of highly skilled IT and analytics staff."

In addition to the bundled software and service sales offered by the likes of Oracle and IBM, business analytics services in the consulting domain generated several billion dollars more worldwide. While the large firms command the lion's share in the analytics market, the billions left at the bottom are still a large enough prize to take the analytics plunge.

Several billion reasons to hop on the analytics bandwagon

While the IBMs of the world are focused largely on large corporations, the analytics needs for small and medium-sized enterprises (SMEs) are unlikely to be met by IBM, Oracle, or other large players. Cost is the most important determinant. SMEs prefer to have analytics done on the cheap while the overheads of the large analytics firms run into millions of dollars thus pricing them out of the SME market. With offshoring comes the access to affordable talent in developing countries who can bid for smaller contracts and beat the competition in the West on price, and over time on quality as well.

The trick therefore, is to beat the IBMs of the world in the analytics game by not competing against them. Realising that business analytics is not a market, but an amalgamation of several types of markets focused on delivering value-added services involving data capture, data warehousing, data cleaning, data mining, and data analysis, developing countries can carve out a niche for themselves by focusing exclusively on contracts that large firms will not bid for because of their intrinsic large overheads.

Leaving the fight for top dollars in analytics to top dogs, a cottage industry in analytics could be developed in the developing countries that may strive to serve the analytics need of SMEs. Take the example of the Toronto Transit Commission (TTC), Canada's largest public transit agency with annual revenues exceeding a billion dollars. When TTC needed to have a large database of almost a half million commuter complaints analysed, it turned to Ryerson University, rather than a large analytics firm. TTC's decision to work with Ryerson University was motivated by two considerations. First the cost; as a public sector university, Ryerson believes strongly in serving the community and thus offered the services for gratis. The second reason is quality. Ryerson University, like most similar institutions of higher learning, excels in analytics where several faculty members work at the cutting edge of analytics and are more than willing to apply their skills to real life problems.

Why now?

The timing had never been better to undertake such an endeavour on a very large scale. The innovations in Information and Communication Technology (ICT) and the ready availability of the most advanced analytics software as freeware allows entrepreneurs in developing countries to compete worldwide. The Internet makes it possible to be part of global marketplaces with negligible costs. With cyber marketplaces such as Kijiji and Craigslist individuals can become proprietors offering services worldwide.

Using the freely available Google Sites, one can have a business website online immediately at no cost. Google Docs, another free service from Google, allows one to have a web server for free to share documents with collaborators or the rest of the world for free. Other free services, such as Google Trends, allow individual researchers to generate data on business and social trends without needing subscriptions for services that cost millions. The graph below is generated using Google trends showing daily visits to the websites of leading analytics firms. Without free access to such services, access to the data used to generate the same graph would carry a huge price tag.

Similarly, another free service from Google allows one to determine, for instance, which cities registered the highest number of search requests for 'business analytics'. It appears that four of the top six cities where analytics are most popular are located in India, which is evident from the following graph where search intensity is mapped on a normalised index of 0 to 100.

The other big development of recent times is freeware that is levelling the playing field between haves and have-nots. In analytics, one of the most sophisticated computing platforms is R, which is available for free. Developers worldwide are busy developing the R platform, which now offers over 3,000 packages for free for analysing data. From econometrics to operations research, R is fast becoming the lingua franca for computing. R has evolved from being popular just amongst computing geeks to having its praise sung by the New York Times.

R has also made some new friends, especially Paul Butler, a Canadian student who became a worldwide sensation by mapping the geography of Facebook. While being an intern at Facebook, Paul analysed gigabytes of data to plot how Facebook's friends were linked globally. His map (see the image below) became an instant hit worldwide and has been reproduced in publications thousands of times. If you are wondering what software Paul used to generate the map, wonder no more, the answer is R.

R is fast becoming the preferred computing platform for data scientists worldwide. For decades the data analysis market was ruled by the likes of SAS, SPSS, Stata and other similar players. R has taken over the imagination of data analysts as of late who are fast converging to R. In fact, most innovations in statistics are first coded in R so that the algorithms become available to all immediately and for free.

Source: The Popularity of Data Analysis Software | r4stats.com

The fact R is freely available should not be taken lightly. A commercial license of a similarly equipped version of SPSS may cost up to US$7,500. The other big advantage of using R is the fact that thousands of training documents on the Internet and videos on YouTube are also available for free by volunteers.

Where to next

The private sector has to take the lead for business analytics to take root in developing countries. The governments could also have a small role in regulation. However, the analytics revolution has to take place not because of the public sector, but in spite of it. Even public sector universities in developing countries cannot be entrusted with the task where senior university administers do not warm up to innovative ideas unless they involve a junket in Europe or North America. At the same time the faculty in public sector universities in developing countries is often unwilling to try new technologies.

The private sector in developing countries may want to launch first an industry group that takes upon the task of certifying firms and individuals interested in analytics for quality, reliability, and ethical and professional competencies. This will help build confidence around national brands. Without such certification, foreign clients will be apprehensive to share their proprietary data with individuals hidden behind computer monitors thousands of miles away.

The private sector will also have to take the lead in training a professional workforce in analytics. Several companies train their employees in the latest technology and then market their skills to clients. The training houses would therefore also double as consulting practices where the best graduates may be retained as consultants.

Small virtual marketplaces could be setup in large cities where clients can put requests for proposals and pre-screened, qualified bidders can compete for the contract. The national self-regulating body will be responsible for screening qualified bidders from its vendor-of-record database, which it would make available to clients globally through the Internet.

The IBMs of the world see the analytics market to hit hundreds of billions in revenue in the next decade. The abundant talent in developing countries can be polished into a skilled workforce to tap into the analytics market to channel some revenue to developing countries while creating gainful employment opportunities for the educated youth who have been reduced to making cold calls from offshored call centres.

Murtaza Haider, Ph.D. is the Associate Dean of research and graduate programs at the Ted Rogers School of Management at Ryerson University in Toronto. He can be reached by email at [email protected]

 

[ani82v]

Mu Sigma surfs the big data wave

A month or so ago, with little fanfare, Mu Sigma, a data analytics company, reached the $100 million high water mark in revenues. Around the same time, it also became a ubiquitous presence in the US, sponsoring and speaking at several important industry conferences such as Enterprise 2.0.

Seemingly out of nowhere, this Northbrook, Illinois-based company that provides decision science and analytics solutions, with a large offshoring centre in Bangalore, has become not just one of the hottest young technology companies around — its success has also become a beacon for what the next big wave in outsourcing will be.

In general, arts and science never see eye to eye. The debate on whether Leonardo da Vinci was a scientist or an artist often raises temperatures. However, when Dhiraj Rajaram started Mu Sigma seven years ago, he was not just creating a company that would provide decision science, but also one that would have both arts and science capabilities.

Mu Sigma is just that, says Rajaram; in addition, it has scale. Art, according to Rajaram, represents design capability and science represents the laboratory. Scale is the ability to work like a factory. "All this is coming together for the first time. Not only in India but anywhere else in the world," adds Rajaram.

'Big data'
Every day, we create 2.5 quintillion bytes of data and 90 per cent in the world today has been created in the last two years alone. This data comes from everywhere: Sensors used to gather climate information, posts on social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals, to name a few.

The problem is, unstructured data — text, videos, tweets and social media — has grown at phenomenal rates. Google and Facebook, for example, are the biggest repositories of consumer behaviour data. If companies can find valuable insights within this seemingly unintelligible torrents of bits and bytes, they will be able to make more informed business decisions, whether in marketing, sales or practically any functionality within a company. This is what Mu Sigma does in its back office in Bangalore: It sifts through mountains of information and analyses it for clients.

"People are looking to make better decisions. The amount of data that is required to create this decision is exploding every day. To make this happen, one needs to have an inter-disciplinary talent of mathematics, business and technology. Every player wants to be able to do this but they do not see all these three things together," he says.

For instance, Mu Sigma helped a New York-based online clothing store spot shopping trends among its customers that in turn radically influenced its website architecture and merchandise selection. Its data analysis skills have already attracted a range of clients, from Dell to large insurance companies to Microsoft's Bing division.

Facilitating more sound business decisions is, not surprisingly, good business. According to a recent IDC research, business analytics grew at a compound rate of 9.8 per cent per year, hitting $35.1 billion this year and projected to reach $50.7 billion by 2016. For 2011, analytics underwent a 14.1 per cent increase in revenue. Mu Sigma impressed enough to attract a total of $150 million in private equity so far — in December last year alone, General Atlantic and existing investor Sequoia Capital announced that they were pumping over $100 million into Mu, while Accel, an early investor exited its stake.

Acting on a hunch
While it may seem that Rajaram has hit the jackpot, he says that when he started Mu's business plan (2003-04), data was not as big it is now. "It was a hunch that this is where the world will be. And in the world of tomorrow, knowledge and learning will be far more important than knowing," he adds.

Rajaram, a engineering graduate from Anna University, Chennai, and an MBA from the University of Chicago, blames his decision to quit a comfortable job at Booz & Co for what he calls 'middle-aged men menopause syndrome.' Quitting his job was not enough. He sold his home in Illinois to raise initial funding. "I realised that calling people and telling them that you want their business and what can you do for them is the easiest way to get work. I did not have any great contacts in the industry, was just 29 and deployed all my savings of around $2.5 million into this company. We were talking to 20-30 prospective clients. And Microsoft happened to say 'yes' first," he reminiscences.

Back to school
Since scale is a critical part of his business plan, India was an integral part, too. "India is not an after-thought. India is a fore-thought for us. This company could not have been built without India. You need scale and you need to train a lot of people in arts and science. People who understand English but are not afraid of Mathematics," explains Rajaram.

Scaling up has meant hiring employees — Mu Sigma has around 1,700 of them. According to a recent McKinsey study on Big Data, India will require nearly 100,000 data scientists in the next couple of years. Rajaram realised early on that he would have to invest in creating a talent base. Doing so entailed methods that were a little out of the ordinary, to say the least.

A candidate applying to Mu Sigma needs to clear four stages of recruitment and a training program that consists of three parts. Under recruitment, the first stage consists of an aptitude test that has basic arithmetic and logical problems. This is followed by a group discussion, "but our group discussion aims to get people out of their comfort zone. While we want people do to the maths, we also want them to be able to talk," adds Rajaram. The third stage has a video synthesis, where a candidate is shown a video and then provided a piece of paper slightly bigger than a visiting card. On it, the candidate has to synthesis whatever he or she has just seen. The fourth is the interview, which the company calls a 'Fit interview'. Of the 100 candidates who apply, only a handful get selected.

Then comes the next big hurdle: passing what the company calls Mu Sigma University, This has three parts. First, the candidate is introduced to consulting principles. The curriculum of the second part consists of analytics, which covers areas like artificial intelligence, business intelligence platforms of SAP and Cognos. Finally, a mini MBA course has to be negotiated. "The entire process takes two to three months. This year, around 1,000 people will pass through the Mu Sigma University. We are creating a new breed of skill sets in the industry. We are changing the world, just like TCS and Infy did in 1990," says Rajaram.

Detractors
Whether it can become the next Infosys depends on how successfully one can sustain this model, which Rajaram says is not just about cost arbitrage. "What we are doing cannot be even termed as outsourcing. That is why it is interesting, as it is a category-defining company," he said.

Not everyone agrees. In an online article titled 'Can Big Data Be Outsourced? Mu Sigma's $150 Million in VC Backing,' Peter Skomoroch, a principal research scientist at LinkedIn, focusing on building data-driven products, as well as a founder of Data Wrangling, which offers consulting services for data mining and predictive analytics, says, "I'm sceptical of the idea of end to end 'analytics outsourcing' right now." Skomoroch feels that vision and creativity are unlikely to be commoditised any time soon. "The competitive advantage in this latitude will go to companies that establish unique data sets and build teams that are aware of how to leverage them. The most plot-changing analytics is going to happen from a small set of talented individuals, not an army of contractors," he adds.

This won't deter the ambitions of Rajaram, however, which are considerable. "I want a little bit of Mu Sigma in every company. Like Intel Inside," he says.

 

[sorcerer]

New frontier opens for Indian techies: Fighting for the nation with data

If you are a techie, your future can be more exciting than writing code for a labour-arbitrage company that sells your skills cheap to the western world. Now you can even fight for your country while not moving an inch away from your keyboard.

A number of tech startups, as well as big companies, are now providing intelligent solutions to soldiers guarding the borders. The demand for such solutions has increased after rising hostilities with Pakistan last year and the Doklam standoff with China this year.

Speaking at the second edition of the ‘Smart Border Management’ conference organised by FICCI in New Delhi on Monday, BSF director-general KK Sharma said technology was required to reinforce the efficiency of soldiers guarding borders.

Border management has emerged as a crucial area in need of smart solutions for armed forces. Most of India's borders are difficult to guard due to mountains, rivers and other diverse geographical features.

That's where Indian techies come into picture. A techie sitting in his Bangalore office with loads of data can devise smart solutions for soldiers guarding the borders. And a lot of them are already working on what has come to be called defence analytics.

With the help of artificial intelligence and Big Data, soldiers patrolling the border know when and where terrorists are likely to cross over into Indian soil. A smart solution devised by a techie can be as precise as predicting the day and time of the next incursion and how it would occur.

"We have more than 20 terabytes of data on border movement which earlier used to be recorded in physical logbooks of soldiers. These include thermal images, instances of people going near the fence from across the border, activity at late night, etc,” says Tushar Chhabra, co-founder of Gurgaon-based driverless truck company CRON systems, which helps the army in predicting border-infiltration patterns.

For instance, terrorists on the other side of the border usually send a herd of cattle to check for mines that the armed forces have placed on the border. Analysis of past data has shown that typically an incursion follows 12 days later.

Nyokas, a Kochi-based startup, has developed a smart jacket for armed forces. When a solider gets injured during an operation, the smart jacket sends signals to other soldiers.

Delhi-based Innefu Labs provides analytics services to BSF, CRPF and police departments across the country. The firm’s data analytics predicted that incidents of cross-border infiltration and narcotic smuggling increase on cloudy and foggy days.

Tata Power SED and Dat Con are running a pilot project of the Ministry of Home Affairs to install an integrated border-guarding system to test technology for preventing infiltration, especially by detecting cross-border tunnels as well as possible entries through aerial and underwater routes. Called the Comprehensive Integrated Border Management System (CIBMS), it is a major counter-infiltration measure to prevent cross-border terror attacks.

CIBMS will establish a seamless multi-tier security ring at the border using a variety of sensors such as thermal imager, radar, aerostat with EO payload, optical fibre intrusion detection system, unattended ground sensor and underwater sensor.

Just a decade ago, artificial intelligence and Big Data were part of arcane lingo of nerds. Today, they have become buzzwords. They are used by entities as diverse as tax sleuths and stock traders.

As ITBP DG RK Pachnanda said at the FICCI conference, the armed forces need technology as a force multiplier. This opnes up vast space for techies. Globally, defence analytics is a $2-billion market but is still taking off in India. While DRDO has a dedicated lab, the Centre for Artificial Intelligence (CAIR), to develop defence technology such as robots, the scope for private data analytics companies is also increasing by the day.
http://economictimes.indiatimes.com...the-nation-with-data/articleshow/60749458.cms

 

[sorcerer]

Social media trends to be analysed to ensure security at airports, nuclear bases | India News

NEW DELHI:

Social media trends and data analytics will be used for the first time in ensuring and monitoring the security at the country's most vital assets such as airports and nuclear and aerospace installations.

The Central Industrial Security Force (CISF), the national agency tasked to secure these important assets, has created a maiden 'media lab' and social media monitoring control room - the Pattern Research For Institutional Social Media (PRISM) analytics - at its base in Arakkonam near Chennai.

A special team of CISF agents has been trained in the job of tracking social media trends, news, reports and indicators across various platforms, collate them and dish them out as vital pieces of "actionable intelligence" to its various airports and other vital units.

Platforms like Twitter, FaceBook, YouTube and Flickr will be used by the paramilitary force to check any suspicious and sabotage-like activity against the assets it guards.

The platform, developed by the IIT Delhi, is used by the Intelligence Bureau (IB) and the Mumbai police to keep a check on security issues.

"We are doing this on an experimental basis. The PRISM control room is based in the southern part of the country as we have a sizeable number of units that have our armed security cover in that part. Based on the experiences, this smart centre will be further bolstered," CISF Director General (DG) O P Singh told PTI.

He said that a special team of his men and women will keep a "track of social media trends" with the sole view of keeping their units such as airports, aerospace stations and nuclear bases safe.

"This is completely in-house. We intend to link it to our WAR (web analytics and resolution) room that is already operational and based in Delhi," Singh said.

"The digital chatter in the world of social media is a great source for intelligence collection and timely action. Public sentiment towards the force, behavioural pattern and predictable actions can be a great help to take timely action and avoid any mishap," a senior CISF officer said.


A red flag would be raised, he said, in the case of any posts that indicate any threat towards the units covered by the CISF or any potential harm that a force personnel would inflict upon self or others.


"The potential of this unique platform would be to avert even suicides in the force, to an extent as any sort of abnormal behaviour or posts would be an indicator for action to be taken.


"This would be done through features like sentiment analysis in-built in the software itself," the official said.


The 1.80 lakh personnel-strong CISF is tasked to secure 59 airports and other vital installations under the command of the Union home ministry.


https://timesofindia.indiatimes.com...rports-nuclear-bases/articleshow/61171210.cms