How Paytm Uses Tech to Manage 200 Million Users

Key points:

- Paytm processed 1 billion transactions in the quarter ended March 2018
- The firm employs 200 product managers and over 700 engineers
- Its data science lab in Toronto, Canada, develops key tech tools
- App analytics and machine learning are used to retain users and for up-selling

Mobile wallets--mobile apps used to pay for recharges, groceries and other daily items--may have come of age in an increasingly digital India, but much goes behind-the-scenes to keep them working well and users hooked.

Paytm, which has 200 million monthly active users and processed close to 1 billion transactions in the quarter ended March 2018, is a case in point. It competes with MobiKwik, FreeCharge, PhonePe and several others in this space.

Discussing the tech strategy at the company in a recent interview, Deepak Abbot (pictured), senior vice president of One97 Communications Ltd, which owns and operates Paytm, said, “Though a payments firm, we are a technology company at the core and everyone here, including Vijay, is a hardcore techie--he even calls mid-level engineers sometimes to discuss architecture design.” (Vijay Shekhar Sharma is the chief executive of Paytm.)

Abbot said that most in top positions at the company either have technology background or are “quite comfortable” with tech. “Culturally, we have a tech mindset. That is another reason we have been able to build a very complex product in a flexible way.”

Sharing insights into what goes on ‘under the hood’ as they say in tech, Abbot said that quick decision-making and a product-centric approach drive software development. “In our meetings, once an idea is crystallized, Vijay is very clear about what product to build. As a result, the product managers are also clear how to get it done. And when the engineers are given very specific details, they are able to quickly build it,” he revealed.

The simplicity of the Paytm app belies its complex architecture and the number of people that work on it. For instance, Abbot said that there are as many as 200 product managers and 700-800 engineers working on different aspects of the app.

But how does Paytm define a product? “At Paytm, a product is defined as anything a consumer—be it is an end consumer, a merchant or a marketplace seller--interacts with,” said Abbot. For example, recharge is a product in itself. Paytm’s implementation of Unified Payments Interface (UPI), again, is a product (UPI is an easy, instantaneous payment system built by the National Payments Corporation of India or the NPCI). “And then you build use-cases on top of UPI such as P2P, P2M and B2B payments. Wallet--the most used product of Paytm--is another,” said Abbot. (P2P, P2M and B2B stand for person-to-person, person-to-merchant and business-to-business respectively.)

The idea of keeping all these products within the same Paytm app, according to him, is that users should move from one product to another seamlessly—something that requires “a highly scalable product” to be built.

Integration of multiple products within the same app also helps Paytm cross-sell more easily to customers, who may first use one product before being “nudged towards” others, said Abbot.

Talking about stickiness of the app and up-selling to users, he said, “We have observed that if a customer has only used Paytm for recharge, then the retention rate for such a user is 40% after three months. But if we can upgrade him to send money to others, they become power users of Paytm and the retention improves dramatically to 70%.”

Industry experts forecast bright days ahead for mobile wallets. The number of mobile wallet users is expected to grow from the current 200-250 million to around 500 million in the next couple of years, according to Probir Roy, co-founder of Paymate and an independent director at Nazara Technologies. While he believes that “the next big thing” will be “interoperability” among different wallets, he noted that it is a tough space to operate in and some consolidation is “bound to happen” in the coming years. “My guess is that the top two or three companies will have 80% of the market,” he concluded.

------Paytm Labs: Managing customer lifecycles-----

To make the most of app analytics that capture user behaviour, Paytm’s data science lab, Paytm Labs, in Toronto, Canada, works on developing multiple software tools. One such key tool is CLM or Customer Lifecycle Management.

According to Deepak Abbot, senior vice president of the company who is based at Paytm’s headquarters at Noida near Delhi, what CLM does is “catch every ‘signal’ from the app”. Explaining how it works, he said, “If you use the app for UPI, it segments you as a UPI user; if you do a recharge, it marks you as a recharge user. It also upgrades you automatically based on your behaviour or purchase history. So, for instance, if you make an electricity bill payment or a post-paid bill payment, it upgrades you to a post-paid user.” There is a lot of granularity built into the CLM tool to classify and reward different levels of users at different times.

The tool puts users in different segments and generates actionable triggers accordingly. “For example, if a premium user who earlier made a money transfer of Rs 5,000 has not used the app for a month, he will be shown a cashback offer or an ad on Facebook,” said Abbot. Similarly, alerts are shown for soon-to-expire mobile recharges and other bills. “The CLM tool uses such alerts and offers to get those customers back into the app. And if they are already in the app, it will customise the view for them by showing up frequently used icons upfront and hiding others,” he explained.

The entire user data in the Paytm app goes into a “data lake”, and the team in Canada uses it to formulate the rules of the risk engine and other software. The data lake, Abbot explained further, is a repository of multiple sources of data, including phone usage data, hardware data and address book; then there is transactional data plus the behavioural data (where the users navigate inside the app, how much time they spend shopping, etc). All this data is used through machine learning (ML) algorithms so that the alerts and promotions can be automated and personalized.

The Toronto team comprises 70 data scientists and engineers and, besides the CLM tool, has developed the company’s risk and customer score engines. “We just plug those products here (in India) and start using them,” said Abbot.

---##----

(Note: An edited version of the above post first appeared on www.livemint.com - where I used to work until recently. The interaction with Deepak Abbot took place during my Mint tenure.)

Telcos Undergoing Transformative Changes Due to Surging Data Demand

On 2 September 2009, The New York Times published an article headlined ‘Customers Angered as iPhones Overload AT&T’.

Calling the new iPhone 3GS a “data guzzler”, it went on to describe how the device choked up bandwidth on the telecom operator’s network, resulting in “dropped calls, spotty service, delayed text and voice messages and glacial download speeds”.

This is just one among several such instances of how telcos worldwide have been struggling to keep up with the burgeoning demand for data services. And much as they are trying, the demand surges keep happening in one or other part of the world (India being an apt case in point at the moment).

Telcos are fighting this battle on two counts. On the one hand, they have been upgrading their mobile networks from 2G to 2.5G to 3G to 4G. And, on the other, they have been deploying various information technology (IT) tools to operate more efficiently, reduce customer churn (customers migrating to other telcos) and to serve customers better.

The woes of telcos are not difficult to discern. From providing plain old voice telephone services up until the 1980s, operators now have to also provide text messaging, multimedia messaging, video on demand, gaming, music and several other value-added services on a mind-boggling variety of handsets.

In fact, the demand for data services is far outstripping that for voice services and causing major structural changes to the business models of telcos.

According to a report by Cisco Systems Inc., mobile data traffic will grow at a compound annual growth rate (CAGR) of 53% between 2015 and 2020, crossing 30 exabytes per month by 2020 (1 exabyte = 1 billion gigabytes or GB as it is popularly known. It is said that 5 exabytes of storage space will be taken up by all the words ever spoken by mankind).

One of the key factors in that data growth is the global popularity of smartphones to access the Internet, watch videos, consume news and other content, connect on social media or even plug into work-related applications such as email, analytics tools and customer relations software.

While much of the investments telcos are making goes into acquiring spectrum and upgrading their existing 2/2.5G networks to 3G and 4G, they are also investing significant amounts in their back-end systems that help them run those networks, including network-monitoring tools, billing software, customer experience management (CEM) solutions, etc. According to estimates by Analysys Mason, a research firm, CSPs will spend over $100 billion per year on software and related services by 2020.

In this context, India is one of the emerging market hotbeds where intense competition is playing out in the telecom market, especially for the relatively more lucrative and faster growing data segment. The latest salvo was fired in September by Reliance Jio Infocomm Ltd, the latest entrant in the country’s crowded mobile communication space. The company claims to have signed up 16 million subscribers in the very first month of the launch, touted to be the fastest such milestone anywhere in the world.

Among other things, one of the biggest competitive edges Jio has, as far as technology is concerned, is that its network is fully based on Internet protocol (IP), the same one using which all computing devices—from tiny smartphones to large web servers in data centres—connect to the Internet. Having an all-IP network allows a telco to use the same underlying infrastructure for voice as well as data and be more agile in terms of market offerings—which is why even voice can be considered just another app on Jio’s network.

Other telcos, in contrast, have a mix of IP networks and the traditional circuit-switched networks in the circles they operate in (India is divided into 22 telecom circles or geographically segregated service areas). From the vantage point of an all-IP telco, their operations would be more complicated and clunky.

That is not to say that telcos with a mixed network set-up are going to scrap their past investments in 2G and 3G technologies: instead, they will compete by optimising their multiple networks and invest in IT tools that allow them to be more efficient and agile.

According to Ekow Nelson, region head, IT and cloud, Ericsson India Pvt. Ltd, “Some of the telcos are looking for a radical transformation of their business in order to look like a digital enterprise. This is a complete transformation of their relationship with their suppliers and customers. Others are looking towards more incremental changes. There is a whole range of different approaches that the operators have and, of course, some of it is driven by where they see themselves (in the foreseeable future).”

“Part of the transformation comes from understanding that this is really about changing the way you approach and interact with your customers and changing the way you organize yourself,” says Nelson, referring to the digital transformation challenges for IT decision-makers at telecom operators.

For example, according to him, if a telco’s distribution channel is through shops and retailers, that is not digital. “A lot of young people buy services online and they want help online. So if you want to become a digital player, then most of your own operating model will have to shift: you need to build online capabilities that allow your customers to interact and operate with you in a way that is very different from walking into a shop.”

He believes that just as the music industry moved from buying and renting CDs to online audio streaming, so is the telecom sector shifting from buying recharge coupons to self-service portals and apps—that is, a digital distribution model. In the case of India, however, a hybrid model that optimises both physical and digital sales for different geographies and customer profiles looks more likely.

Given that roughly one-third (31.3%) of India’s population, according to the Census of India 2011, is in the age group of 18 to 35—a generation cohort more digital-savvy than the rest—telcos that build a greater connect with them can reap significant business benefits. And one tried-and-tested way to do that is to app-ify most of their offerings and throw as many customised pricing plans at them as IT agility allows them to.

An indication of the importance of an app-driven approach is the recent marketing campaign of Bharti Airtel Ltd, India’s largest telco with an India subscriber base of over 250 million. The ad shows how quickly the new and integrated MyAirtel app can be downloaded onto a smartphone.

Earlier, there were several apps for music, movies, money, news, etc. but the new app comes as an integrated bundle (Reliance Jio’s MyJio app, which launched before Airtel’s new app, works in a similar fashion).

According to Animesh Sahay, senior country director of sales (enterprise and telecom business), CA Technologies India Pvt. Ltd, a provider of enterprise software and services, “For telcos, it is becoming increasingly important as to how they can wrap the entire app in a fashion that they are able to record the customer experience. Today, if a customer has a bad experience with an app, they might give it a try twice or thrice, but after that they are just going to junk it.”

So it becomes very important to know what the customers are experiencing on the app and to get their feedback and tie it back to app development, he says.

A telco can install an app tool to have a view of exactly what the customer is doing, exactly where he had an issue, what the screen looked like when a particular transaction was happening on the app, etc.

In short, the tool allows the telco to replay the same series of steps the customer took and find out what went wrong and where.

Another thing operators need to do, according to Sahay, is to move away from the old, waterfall model of application development to agile development methods by embracing what is called DevOps. DevOps is the combination of development (Dev) and operations (Ops), referring to how the IT teams at most enterprises are divided.

Traditionally, there has been some friction or lack of coordination between the two teams that typically work in isolation. The DevOps movement calls for a greater cohesion between the two and the use of agile software methodology and tools that enable it.

The whole idea of DevOps and agile method is to release newer versions of software or apps as quickly as possible so that new features and benefits could be marketed to existing and potential users.

In addition, given the speed at which mobile technology is moving today, more and faster releases help fix multiple bugs and issues with the software.

The dynamism in the telco universe is causing many to move towards what is known as a catalogue-driven architecture, which allows a telco to dynamically serve up data plans and other service offerings (movie/music downloads, for instance) to customers even if third-party mobile valued-added services providers are involved.

Going forward, most telcos in India, including Airtel, Vodafone, Idea and others, will ramp up their digital transformation efforts to increase data revenue and stay relevant in a fiercely competitive market.

(This post first appeared in Mint: http://bit.ly/2fv9srl)

Disaster Management: Can Internet of Things Make a Difference?

What happens when disaster strikes? The answer depends, among other things, on where you are located. And if you live in a third-world, hot, crowded and messy country like India—all hell breaks loose.

Millions among India’s billion-plus citizens have seen that hell from up, close and personal: in the ruthless form of floods, earthquakes, cloudbursts, landslides and other disasters that destroy lives, livestock and the lock, stock and barrels that help people sustain their existence.

In fact, as I write these lines, the country is in the midst of disastrous rains and flooding in several states across its length and breadth.

On such occasions, the administration goes into an overdrive, the army and paramilitary forces are called in and the voluntary organizations are roped in for relief work. But Nature’s fury often proves too much and, despite all their efforts and hard work, the scale at which misery unfolds in the aftermath is astounding.

Can technology play a role in anticipating, mitigating, controlling and managing this misery? And if so, to what extent and in what ways?

Those were the questions that came flooding to my mind as I attended the launch recently of a white paper titled “Internet of Things (IoT) for Effective Disaster Management.” The paper was brought out by Digital India Action Group (DIAG), a think tank set up by IT vendor lobby group MAIT for “ideating and monitoring policy initiatives to support the Indian Government’s mission of Digital India.”

The objective of the paper is “to create awareness and appreciation about the potential use and applications of IoT for different aspects of disaster management.”

Alongside, DIAG also released another white paper, “Aadhaar-Enablement: A Framework for Citizen-Centric Services”.

For the uninitiated, Aadhaar is a 12-digit unique identity issued by the Government’s Unique ID Authority of India. Over 1 billion of these IDs have been given thus far in what is billed as the largest such exercise in the world.

While discussing Aadhaar and the potential of Aadhaar-based services is a Pandora’s box in itself, let me confine myself to IoT in disaster management for this post.

The role of IoT in disaster management, in keeping with the huge potential of this mother-of-all-technological-paradigm, is critical and wide-ranging. A multiplicity of agencies, infrastructure, devices, policies, and applications, among others, must come together to make the whole exercise “effective”, as the DIAG paper rightly highlights in its title.

The presence of a cross-section of officials and executives—from government, industry and consulting organizations (see pic)—is, one hopes, symbolic of the coordinated, on-ground effort that will be required in the days to come to give actual shape to the vision laid out in the document.

The IoT white paper recommends a “Seven-Point Action Plan” to shift from a “relief and recovery” model to “risk and vulnerability assessment” and address key issues and challenges related to management of natural and man-made disasters in India.

According to data from the IoT white paper, as much as 57% land area of India is vulnerable to earthquakes; 12% of this area is vulnerable to severe earthquakes. Besides, 68% land is vulnerable to drought, 12% land vulnerable to floods and 8%, to cyclones. The paper notes that many cities in India are also vulnerable to chemical, industrial and other man-made disasters.

The benefits of IoT in disaster management are easy to visualize (though difficult to implement, given the current realities of India): agencies can gain a clear picture of operations with real-time visibility of data as well as model data from multiple sources. This can further be transformed into accessible, actionable intelligence for faster, better-informed decisions. It is important, therefore, to create “a single, federated information hub.”

The paper calls for building an information backbone which all parties—government agencies, NGOs, infrastructure operators and community—can contribute to and work from.

One term in the paper that specifically caught my eye was “intuitive analytics” which seems to take the capabilities of the current big data analytics technologies to their optimal level.

In this context, SAP’s Lovneesh Chanana presented an insightful video of the city of Buenos Aires in Argentina. After the disastrous floods in the year 2013, which resulted in loss of close to a hundred lives and millions of dollars, the Argentine capital decided to put sensors in over 30,000 storm drains that measure, as per this report on the SAP site, “the direction, level and speed of water.” One of the key technologies to gather and analyze this huge amount of data in real-time is SAP HANA.

Technologies lie SAP HANA (or IBM Watson, for that matter) are not cheap to deploy for funds-starved governments. But consider the impact of not using the most advanced technologies: A World Bank forecast puts the annual losses from floods alone to reach as high as $1 trillion worldwide if cities don’t take preventive measures.

Each city, in my opinion, will need to take a deep view of what’s the best fit for it in terms of technologies, including IoT and the use of social platforms such as Twitter and Facebook. (If you think lightly of the idea, pause for a moment to consider that the US Geological Survey, a government entity, runs a service called the Tweet Earthquake Dispatch (TED). Under this, there are two Twitter accounts that send out earthquake alerts: @USGSted and @USGSBigQuakes.)

I remember reading a report a few years ago that was in a way precursor to the TED service. When, in the US, a 5.9-magnitude earthquake shook the Northeast in 2011, many New Yorkers learned about it on Twitter—seconds before the shaking actually started. Tweets from people at the epicenter near Washington, D.C., outpaced the quake itself, providing a unique early warning system. (Conventional alerts, by contrast, were said to take two to 20 minutes to be issued.)

Technology is advancing at a much faster pace now, especially with machine learning, robotics and drones appearing more frequently in headlines than ever before.

What should the Indian government and industry players be doing in tackling disasters with IoT and other tools?

The DIAG white paper gives some recommendations, the MAIT DIAG Seven-Point Action Plan, which includes:

- Release of cloud security and related guidelines as part of the Digital India policy framework.

- Inclusion of ICT in Disaster Management in the National Skills Development Framework and Plan.

- Release of IoT Policy for India.

- Development of framework for continuous industry participation in planning for disaster management.

- Back-end applications for asset management with disaster management authorities.

- Knowledge portal for sharing experiences and best practices.

- A comprehensive plan for prevention of cyber disasters.

Even if some of the above points are put into practice by a government-industry “action tank” (taking the think part to its logical conclusion), the disasters that certainly, unavoidably await the Indian multitudes can perhaps be mitigated and managed much better than before.

For CIOs, tech leaders and others who would like to dig deeper or get involved, here are some reference links:

Revised Draft IoT Policy of the Government

National Policy on Disaster Management

India Disaster Resource Network

UN ESCAP: ICT and Disaster Risk Reduction

 (The above blog post first appeared on dynamicCIO.com. Lead visual credit: Pixabay.com)

The Big Data Tech behind Times Internet’s Native Ad Play - Colombia

Not many people may know it, but one of the largest publishers in India, The Times of India Group, is also home to one of the largest publisher-owned ad network platforms in APAC. The Group’s digital venture, Times Internet Ltd (TIL), runs one of the most complex and sophisticated ad serving operations, in addition to hosting the editorial content for multiple sites (internal as well as partnerships).

Around a year back, TIL had launched its own native ad platform, called Colombia. (Native ads are personalized ads that are shown to the web users based on their past browsing history, interests, etc., and are usually text ads as opposed to display/banner ads.)

Given the growing global trend of more and more brands putting their money on native ads, platforms such as Colombia are gaining increasing significance in the media world. The platform ensures similar user experience across mobile and web.

I recently spoke to Sumit Malhotra, Head – IT, TIL, to know more about how they developed Colombia, the technology behind the platform and challenges associated with what Sumit calls a “big data recommendation system.”

It would be pertinent to note in this context how the TIL ad network has grown in the past two years, which is nothing short of exponential—from serving around 40 million ad impressions per day in last June to a peak of 500 million impressions/day this month.

Besides serving ads from its own marquee properties, TIL has tie-ups with third party ad networks like Taboola and many others. So the ad inventory that is served through Colombia comes from a number of sources, all of which need to be integrated tightly with the TIL platform for serving to the user—who could be sitting and surfing in any part of the world. 

The key, says Malhotra, is to provide a consistent experience to the audience with as low latency as possible (often, the native ads are served in 100 to 150 milliseconds but, in any case, the threshold has to be kept below 500 ms).

“Otherwise, the user will have either scrolled down the page or moved on to another site (without seeing or clicking on the ad),” he says.

The biggest challenge for any ad network today is to deliver an ad at a low latency. To do so, it is important that calculations and permutations related to which ad is to be served to which user profile are based on the recommendation of the big data system—Colombia in TIL’s case—which is run in-memory rather than on disk.

Talking about the challenges, Malhotra says, “Another challenge is that suppose a user is coming from the US and hitting our servers in India, so the travel time for a data packet is quite high; we need to take it closer to the audience. And since every ad is personalized, that is a big challenge.”

Part of the low-latency challenge is solved by having multiple ad server clusters in different geographic regions of the world. TIL has its servers hosted in different geographies and uses a mix of public cloud options and its own data centers. “This helps us serve ad requests within specific regions,” he says.

Another very notable and significant thing is that TIL has custom-built its own big data engine using open source tools and technologies—all done in-house by its 100-plus technology development team.

It took slightly over a year to build Colombia and then another 9 to 12 months to roll it out fully across the board for all online properties.

“Colombia was launched about eight months to a year back. And given that the infrastructure is huge, we had to roll it out to all the properties, 40 different brands across Times Internet. So the deployment also took time, as the technology needed to be integrated with the publishers as well. And after rolling out, you understand the issues and then scale it up slowly,” says Malhotra. Now it is fully deployed not only across all TIL properties but with the third parties as well.

The key benefit is derived from “the complex algorithms that help us run highly targeted campaigns,” he says.

Native ad platforms are one of the primary reasons why today’s users get the sense that if they go to particular kinds of sites (sport, technology, housing, etc.), the ads that accompany on the sites that they visit next are mostly related to the content they had just viewed.

“Given that it’s a personalized ad network, we need to do in-memory data recommendations. For maintaining low latency, we cannot afford to do any calculations on data that goes to the disk, all calculations have to be done in-memory,” says Malhotra.

Malhotra says that most components of big data analytics systems today run on bare metal servers rather than virtualized ones, as the virtualization adds another layer to the process and increases the latency. With the scale and complexity that TIL operates in, he says, “we cannot survive any virtualization overhead.”

On being asked why TIL didn’t go for a branded big data analytics platform (from known large vendors), he had this to say: “Because those vendors cannot give you that kind of personalization in-memory at this rate and at this kind of a size/scale. So for example, if we are using, say, 300 servers for our open source solution, it would require 600 or more servers to do the same things on vendor products.”

Also, Malhotra is of the opinion that if one uses a vendor product, one gets locked on to it and it also takes away the flexibility.

So it looks like TIL is not going to take the outsourcing route in the near future.

To keep its ad network in good health, TIL also has to do a lot of monitoring of how the ads are being served across different geographies. One, to check if a consistent experience is being delivered to the user; and two, whether a campaign is not surreptitiously trying to deliver some malware to the end user’s device (in which case the campaign is stopped and/or blocked.)

For this, TIL uses a mix of in-house resources as well as tools from third-party providers.

The challenge now is that, sometimes, the traffic at multiple sites can peak at the same time, which requires a different kind of scalability.

(Note: This blog post first appeared on dynamicCIO.com)

Leading Data Scientist Talks about, Well, What Data Scientists Do!

Just as data keeps proliferating all around us, there is a great hue and cry about what to do with all those terabytes, petabytes, exabytes…whatever bytes you! Sure, there are ever powerful number-crunching machines and more capable software, but at the end of the day, you are going to need professionals especially skilled in the science of data analysis, management and insights.

That will be the Data Scientist, a role dubbed by some as the sexiest job of this century. Sexy not necessarily in terms of what all it involves but certainly in the high demand and even higher pay packets.

But what exactly would these data scientists do?

An illuminating blog entry on this very interesting and still intriguing question was posted recently by Bernard Marr, an analytics expert and founder of Advanced Performance Institute. To demystify what the work of a data scientist actually involves, and what sort of person is likely to be successful in the field, Marr spoke to one of the world’s leading data scientists, Dr. Steve Hanks—a doctorate from Yale who has worked with companies like Amazon and Microsoft.

Currently the Chief Data Scientist at Whitepages.com (whose Contact Graph database contains information for over 200 million people and which is searched 2 billion times a month), Dr. Hanks talks about some key attributes of a data scientist: One, they have to understand that data has meaning; Two, they have to understand the problem that they need to solve, and how the data relates to that; and Three, they have to understand the engineering (behind delivering a solution).

While all three of these capabilities are important, writes Marr, it doesn’t mean there’s no room for specialization. He quotes Hanks as saying that it is “virtually impossible to be an expert in all three of those areas, not to mention all the sub-divisions of each of them.” The important thing here is that even if one specializes in one of these areas, one at least has good appreciation of all of them. Further, in Hanks’ words: “Even if you’re primarily an algorithm person or primarily an engineer—if you don’t understand the problem you’re solving and what your data is, you’re going to make bad decisions.”

I can especially identify with the “holistic appreciation” quality of data scientists, as many CIOs and development project heads have often shared similar sentiments about most code writers: they are too narrowly focused on the “problem” at hand and usually miss the big picture about the whole project.

Fortunately, unlike the job of a programmer, the field of data science is attracting or likely to attract people “of different personality types and mindsets.”

Having said that, the main challenge for data scientists is not in specializing in a particular machine learning algorithm or a particular sub-field or tool, but in keeping up with the general speed of development in data science, the blog notes.

For more interesting details and insights, I would urge you to read the full blog post.

Do let me know what you think of the fast-emerging field of Data Science.

(Note: This blog post first appeared on dynamicCIO.com. Image courtesy: Americanis.net)

5 Ways CIOs Can Transform Their Companies into Data-Driven Enterprises

Data, data everywhere—not just the right kind to make effective decisions! It wouldn’t be wrong to assume that this is the common lament in most enterprise decision-making circles today.

On the one hand, companies are drowning in an unprecedented flood of data, structured as well as unstructured. And, on the other, CIOs, CMOs and other CXOs are struggling to get a handle on all that data, put it into the right perspective, extract and massage it into a usable form and take quick, effective decisions. The ones that can earn their firm the much-prized moniker of an agile business or a data-driven enterprise.

While making decisions in any enterprise involves a whole battalion of executives, LOB heads, managers, supervisors and many others, I think the job of enabling the whole organization to take decisions based on analytics rather than hunches (and perhaps, lunches) is most suited to the CIO. The reason is simple: who else has an across-the-board view of the data ecosystem of the company? And that too with the additional knowhow of how the information systems work (or can be made to work)?

So, without further ado, here are five ways CIOs can enable an environment for adaptive, data-led decision making in their organizations:

Making speed count: I know one thing for sure: organizations of all stripes today collect all sorts of data. Through all sorts of forms. By making innumerable number of calls to customers and prospects. And by sources such as the usual enterprise data captured through ERP and other operational systems. But how fast are you with the data you collect? Does it lie buried into file cabinets or dusty disks? Simply putting the data to quick use can make a huge difference to the organization. Following up on a hot lead in quick succession of the data collection process, for instance, will translate into revenue; too much delay, on the other hand, will make the prospect turn to your competitors.

Knowing your data from your metrics: This may sound simple to some and unnecessarily complicated to others. Yet this article on the Harvard Business Review site illustrates the difference and the significance of the difference quite clearly. Authors Jeff Bladt and Bob Filbin cite in the article the example of a YouTube video, asking the reader to guess as to how many views would qualify a video as a success. Now, the particular video in question had garnered 1.5 million views but it failed to do what it was supposed to do: encourage young people to donate their used sports goods. So, despite the impressive views, only eight viewers signed up for donation—with zero finally making the donation!

Not all results (or metrics) will turn out to be in such low extremes. But the point is well-made: you need to specify clear metrics in any data collection or numbers related exercise that will reliably give the true measure of success for the initiative.

Data is data is data, right? Wrong: When data is to be put at the heart of decision-making in an enterprise, it matters all the more that the data be accurate, consistent and timely. So, one may be under the impression that all the data required for a project, say, a marketing campaign, is available, if the data quality is not up to the mark, the results of the campaign would certainly be below expectations.

According to a data quality study by Experian Information Solutions, 32% of U.S. organizations believe their data to be inaccurate and further, 91% of respondents believe that revenue is affected by inaccurate data in terms of wasted resources, lost productivity, or wasted marketing and communications spend. If that’s the case with such a data-rich economy, one could imagine how bad the shape of things would be in a country like India, where data collection and research are relatively new fields and far from being mature scientific disciplines. In this context, the need for best practices as well as tech tools in maintaining high data quality cannot be over-emphasized.

Democratization of analytics: How many of you can remember the era of generating sporadic MIS reports for the consumption of the privileged few? Well, that era is long gone. However, most companies are still chary of sharing key statistics or analytics data beyond the confines of top or senior mid-management. But gradually, this state of affairs, too, is set for a bold change. Some call the coming wave as the democratization of data or analytics, in which actionable data percolates to the lowest links in the organizational hierarchy.

Having said that, democratizing data does not mean dropping a huge spreadsheet on everyone’s desk and saying, “good luck,” as Kris Hammond, Chief Scientist at Narrative Science points out in this article. On the contrary, he explains what it involves simply and emphatically: “Democratization requires that we provide people with an easy way to understand the data. It requires sharing information in a form that everyone can read and understand. It requires timely communication about what is happening in a relevant and personal way. It means giving people the stories that are trapped in the data so they can do something with the information.”

Point well made: unless people can take “informative action,” the analytics tools or the extracted data will have little value for the people or the organization.

Analyzed this, have ya? Now visualize that, too: I’m not sure if you noticed but the Internet has been flooded with a new tool of information dissemination in the past couple of years. It’s called the infographic. For most of your searches on Google, there are now an eye-load of infographics, those illustrative diagrams that give you the needed information with icons, pictures, graphs and anything non-text.

Much less noticeable but equally important, a similar movement is underway within enterprises in the context of data analytics. Vendors such as Tableau Software and Qlik Technologies are leading the charge in this emerging segment, referred to as the visual analytics market

According to specialist consulting firm Atheon Analytics, visual analytics “brings together computer science, information visualization, cognitive and perceptual sciences, interactive design, graphic design, and social sciences.” (To see the power of visualized data in action, watch this slightly old but enormously impactful video, the Joy of Stats, of Swedish statistician Hans Rosling, who is often referred to as the “Jedi master of data visualization.”)

The above are only a few of the multiple ways in which CIOs can bring the hidden power of data to the forefront of organizational ability and agility. There are plenty of tools and technologies available but each organization must find its own best-fit path to data-driven success. The key is to start the data journey as early as possible and do so in right earnest.

It's a SMAC, SMAC World

Crazy as it may sound, the entire world is going to be SMAC'ed.

Now, now, I'm not a futurist, not even a technologist. But as one who often writes and talks about tech trends that affect enterprises big time, I can bet a few shirts on this: Social, Mobile, Analytics and Cloud will shake things up like hell. (Or heaven, if you are on top of the stuff.)

It is hard to pinpoint when these four potent forces began to coalesce but, according to an article on Forbes.com, it was Cognizant Technology Solutions researchers who coined the term “SMAC” as recently as 2013. But let's not confuse terminology with technology (I remember one firm had coined a similar but rather ugly “SoMoClo,” for Social, Mobile and Cloud; and there are others who use “Nexus of Forces” and “the 3rd Platform” for similar concepts).

Irrespective of the term, the four mega trends are transforming how technology is developed, purchased, deployed and used. And how employees, partners, clients, customers and other stakeholders behave. It is the combined behavior that is slated to make the most impact—which is why it is difficult to put a dollar value to it and why impact estimates vary so much. To cite but one figure, a Reuters.com report quoting Nasscom director Rajat Tandon says that the value of outsourcing contracts for SMAC is set to soar from $164 billion last year to $287 billion by 2016.

No matter how you cut it, SMAC technologies are slated to leave a lasting impression. A Cognizant report, titled “The Value of Signal (and the Cost of Noise): The New Economics of Meaning-Making,” summarizes the situation neatly. “Nearly every aspect of our daily lives generates a digital footprint. From mobile phones and social media to inventory look-ups and online purchases, we collect more data about processes, people and things than ever before. Winning companies are able to create business value by building a richer understanding of customers, products, employees and partners—extracting business meaning from this torrent of data. The business stakes of “meaning-making” simply could not be higher,” it says.

I often hear murmurs of dissent: “There is more hype than substance to SMAC.” Or, “Big data (or cloud or mobility or social media) is not for us.”

It is possible that one of the big social platforms as we know it ceases to exist one day. Or some term other than SMAC will prevail (like cloud prevailed over service-oriented architecture). But does anyone seriously think people won’t be more mobile going forward? Or the human instinct to extend their socializing to new, emerging media will lose steam? Or, for that matter, we will stop analyzing this and that and what not, for business and for pleasure?

Crazy as it may sound…