Business Critical Line-Of-Business (LOB) Data in SharePoint 2013 – Using Winshuttle for no-code or minimal-code approach
In this part 3 of my blog series, I’ll discuss a minimal code approach to integrate SAP with SharePoint. This approach would mean utilizing a tool or toolset to provide an end-to-end solution. I’ll further discuss how we can use the Winshuttle suite of products to enable integration of SAP with SharePoint 2013.
Using Winshuttle, one can create an end-to-end solution rather than just creating services that expose SAP data. Winshuttle has products that allow you to connect to multiple SAP touch points such as SAP Transactions, BAPI’s, and tables and expose the data as a web service. There are lots of tools in the market that allow you to connect to SAP BAPI’s and tables but Winshuttle is unique that it allows you to record entire SAP transactions such as MM01 to create a material in SAP. This allows a non-technical person well-versed in creating SAP transactions, such as a Master Data Analyst, to create a powerful web service that exposes a complete business process in a matter of few minutes.
A separate set of products from Winshuttle then can be used to not only create an electronic form that can be used to consume the web services but also design the business workflow process using a designer tool. In the end you can integrate most SAP business process without writing a single line of code. For example, you can create a SharePoint portal that can integrate SAP Master Data with SharePoint and further allow users to participate in a workflow.
A vast majority of solutions can be implemented by someone who is well versed in the tool and has good understanding of SAP LOB data. This is a significant benefit as it reduces the constant dependence on IT, and the ability to have more self-service type solutions makes business more agile. As a result, the savings are significant as even the simplest self-service solution can save countless hours that would be spent otherwise in manually entering the data in SAP.
Now anyone who has been involved in creating SAP integration solutions for the Microsoft stack of products knows very well that no two SAP integrations are the same and that no product in the market will be able to solve all the business problems. Ideally you want all your customers to build their requirements around the strengths of an enablement platform such as Winshuttle. However in reality that’s not always the case. Very frequently you will run into issues where a product at-least out of the box is not going to solve all your needs.
This is where Winshuttle is different. Yes, it allows you to create moderately complex no-code solutions but for those advanced application-like scenarios, the platform allows more advanced users such as IT programmers to extend the solution. The person implementing the solutions doesn’t necessarily have to be an IT Programmer but, it will certainly help to have such a person implement more advanced solutions as they will require some custom coding. As a result, extremely powerful SAP data automation solutions can be created using Winshuttle with some minimal coding.
It is no longer acceptable to wait months to build a data warehouse, create reports or add new insights to a dashboard, the world and business are moving too fast. Today, customers already have solutions in place that they are using to provide insights to most of their questions. In fact 83% of data already exist in some reportable format. The rest of the information they are looking for also exists but is typically stored in local databases or spreadsheets. The goal of an Agile BI solution is one that can quickly connect the existing BI solution and these new databases or spreadsheets to deliver new insights.
OLD DOG, NEW TRICK
The traditional BI approach involves defining business requirements at the entity level, building and architecting a data model, configuring an ETL process and then building reports and dashboards over the data. An Agile BI approach starts with the data visualization and leverages existing data sources regardless of format or structure. It leverages the cloud for quick results and focuses on the user experience and collaboration.This Agile BI approach allows users to see results in day or weeks, not months or quarters.
We are not saying you should build your data warehouse on standalone spread sheets or databases, but as new insights are being defined and tested, a more agile approach is required to quickly validate assumptions and deliver results. In the last two years we have seen an explosion in the Line of Business driving business intelligence solutions. In most instances these people have a deep understanding of the data and they have already built interfaces or created extracts for the information they need. The goal when working with these LOB users is not to build the perfect data warehouse. The goal is to leverage what is already in place and deliver results. Most companies have all the traditional reports and dashboards already in place to run the business. The new insights LOB are looking for involve combining the existing solution with new data sources.
Cloud solution, Public and Partner Data Sources are offering a completely new opportunity to uncover new insights. Companies today are finding value by combing existing data with 3rd party or public data. Census data, retail data and consumer data all offer new ways to look at the business and deliver value. New data sources and tools offer opportunities for Line of Business users to validate assumptions and uncover business value by leveraging Agile BI.
Creating documentation is one of the more tedious aspects of the software development process. Not that writing algorithms and object-oriented class hierarchies isn’t tedious, but it often seems (at least to me) that writing plain text explaining your code is more tedious than writing the code itself. Also, if you consider yourself an Agile practitioner (as I do), you’re always evaluating the time spent vs. value added whenever you write much about working software outside of and separate from the working software you already wrote.
I find myself constantly analyzing and re-analyzing the darn thing. Is it too long? Too short? Too wordy? Oversimplified? Too granular? Too vague? Should I have even attempted this?
I don’t claim this conundrum is unique to me. It’s one of the main reasons that many software projects are undocumented or under-documented. Because of this, a tool that helps you write clear, simple documentation, and saves you time doing it, is an invaluable find.
RAML is one of these fabulous creatures.
I was introduced to RAML by a colleague while working on an ASP.NET Web API project over the the last few months. What I like best about RAML is that it solves multiple documentation problems through the creation and maintenance of a single file:
- You can design and modify your API collaboratively without writing / re-writing any code.
- The syntax is simple and to-the-point, reducing time needed to properly describe the routes, actions and parameters.
- This document serves as a team “contract” of sorts throughout the development process. Server-side developers, client-side / mobile developers, and testers – everyone – can see and agree on the intended behavior and content of your API requests and responses.
- When the project is done, you’re left with an excellent artifact to hand off and / or publish describing exactly how your API works.
Now that’s what I call valuable documentation. To get started, all you really need to write a RAML doc is a text editor and access to the spec. However the team over at MuleSoft has developed a superb browser-based editor for RAML, complete with spec references and “intellisense” of sorts. Best of all – it’s free to use. They’ve labeled it as “API Designer” on their site; here’s a sample (click the image for a better view):
I used it for the project I mentioned above; I found it to be far quicker than writing a text-only version. Being able to click on and view the routes on the right side of the screen was also a really handy feature during various design and testing discussions. I found the API Designer to be a solid tool, and highly recommend it.
So the next time you kick off a REST API project, *start* with the documentation – and write some RAML.
The conversation about business outreach (service first) and infrastructure elasticity (cloud) does not feel complete without including…
Every generation of technology upgrade has created a need for the next upgrade in some ways. User created content and social media initially drove the need for big data techniques. However, the drivers to this movement added up pretty quickly because what big data analysis and prediction can do for business was quickly understood.
A Quick Introduction
Big data is commonly referred to as the mining and analysis of extremely high volumes of data. The data in question is not structured since it is collected from a variety of sources. Many such sources might not be following any standard storage format or schema. The data in question is also described by its characteristics, primarily – volume, verity, velocity, variability and complexity. The techniques for analyzing this data involve algorithms that engage multiple processors and servers that distribute the data to be processed in a logical way. Map-Reduce is one of the many popular algorithms, and Hadoop is one of the popular implementations of Map-Reduce.
Big data techniques are not something that only the corporations that collect social media dust need, it is something that every business needs to look into, sooner than later. It is a separate topic that every business needs to factor in social media data in some form or the other. Even if that part is ignored, the volume of structured data is increasing by the day.
Keeping all that in mind, it should be important to explain how well big data fits into the elasticity of the cloud. Imagine an operation where data needs to be segregated by some specific parameter on different servers. These different servers might run some processing depending of the type of the data or just store that data to improve access time. A true cloud environment will be the perfect host of such an operation. You can spin up new servers, having specific configuration, with just a few lines of scripts at run time.
Where are we heading
In 2011 Google Fiber announced Kansas City to be the first to receive 1 Gigabyte per second internet speed followed by Austin, TX and Provo, UT. As per the company’s announcement in February 2014, Google fiber will be reaching to another 34 cities. AT&T stepped up by announcing that its new service, GigaPower, will provide the gigabyte internet speed to cover as many as 100 municipalities in 25 metropolitan areas. Besides Google and AT&T many other large and smaller providers are working on targeted areas to provide superfast internet speed such as – Cox, Greenlight Network, Canby Telcom, CenturyLink, Sonic.Net etc.
Considering this new scale of data on bandwidth, the way application technology works is going to change, specially the part that involves Mobile and Cloud. It will be much more convenient to have a huge memory and processor centric operation running in a cloud environment, streaming status and results to the browser running on your laptop or a small hand held mobile device.
Moving the heavy lifting work to the cloud and keeping the control on low resource devices is not something that is going to happen. It is happening now; only the scale and outreach is going to increase exponentially. Everyone connected to this field should to pay attention to the changes and keep a strategy for the future, be it providers, consumers, decision makers, technology workers, business users and consultants.
Power BI is Microsoft’s cloud based service that leverages Excel to enable self-service business intelligence. The term Power BI has also been used generically to reference the components and technologies that compromise the Microsoft BI suite of tools. Specifically, PowerPivot, PowerView, PowerQuery, PowerMaps, Question & Answer (Q&A) and now Forecasting. The Q&A and Forecasting features are currently supported only in Office 365 and SharePoint on-line. The other features are fully supported in the desktop (Office Professional Plus) and Office 365 versions of Excel 2013.
The latest incarnation of Office 365 implements time series analysis to provide forecasting capabilities. It is this version and its forecasting capabilities that will be discussed in this article. The description and definition of the specific time series algorithms related to forecasting is beyond the scope of this discussion but, the implications of providing this capability are not.
The methods and techniques for time series analysis are well documented and understood in academia and in the field of statistics, but now this capability is being placed in the hands of the masses that may or may not have a thorough understanding of the associated techniques or how to interpret the results. This may present a change management issue for an organization, but with some planning a great deal of benefits and insights can be obtained that would otherwise not be realized.
From a change management perspective, it is imperative that a consistent approach be defined and implemented to ensure consistent results when developing an analytics solution. This should also include a training program on terminology, techniques, methods and practices.
Let’s take a detailed look at the process that will lead to obtaining useful insights from a forecasting exercise and then how this process applies to an example implemented in Power BI.
- Business Understanding – Understanding from a business perspective of the project objectives, requirements and what the specific outcomes should be. This may also include an initial reference to an analytic methodology or approach (forecasting, classification, etc.).
- Data Understanding – Understanding of the traits/personality (profile) of the data. Are there data quality issues? What are the valid domains of attribute values? Are there obvious patterns?
- Data Preparation – Does the data need to be reformatted? How will missing values be handled? What are the relevant attributes or subsets of data?
- Modeling – Identify potential modeling techniques to meet the requirements of the business solutions and its objectives.
- Evaluation – Evaluate the model and determine its fitness for use. How accurate is the model? Does it address the business requirements? Have new insights been exposed that change the understanding of the data?
- Deployment – Present the model results. Make sure the appropriate visualization is used to present the results. Does the deployment require a simple report, or is a new process required to closed the analytic loop?
This process is depicted in the following diagram:
The above process steps define the CRISPtm data mining methodology which provides an excellent foundational approach and process for development and deployment of predictive analytic and data mining solutions. It has been around for some time, but the basic tenets are very applicable. Let’s now look at an example of how Power BI forecasting can be leveraged and how the process steps are implemented.
The following data represents new and used car sales from 2002-2014. The data is stored by month. Examining the raw data, this is the opportune moment to address business understanding and identification of the business problem and requirements. In this case the business problem is to forecast future new car sales to help better manage inventory. Also, understanding the nature and characteristics of the data should be accomplished at this point. This can be done via data profiling (min, max, null counts, standard deviation, etc.) and through data visualization. It would also help to have a domain expert available to provide additional insights. With regards to data preparation for Power BI Forecasting, there should be an attribute that can be used for time series analysis. In this case, a new attribute is created named [Period Ending] that is a combination of the [Year] and the [Month] represented internally as a date.
The above data was loaded into a PowerPivot workbook and uploaded to Power BI where some visualizations were applied. The line chart shows new car sales units over time. This line chart will be our candidate for time series analysis (forecasting). Note that there appears to be a cyclical pattern in the data. This is a good reason to generate a visualization to provide insights into the nature of the data.
Currently, to perform forecasting, Power BI must be placed in HTML5 mode. This is accomplished via an icon in the lower right corner of the web page. Once that has been done, then hovering over the chart will expose a caret that indicates forecasting may be performed.
Clicking the caret produces a forecast and displays an additional panel that contains adjustable sliders for confidence interval and seasonality. The forecasting algorithm will attempt to detect seasonality and display the calculated cycle in terms of units. The seasonality slider allows for manually setting the number of periods over which cycles will repeat. For example, If based on domain knowledge, it is known that the seasonality is different from what is calculated then it can be adjusted accordingly. This may change the forecasted values. In this case, the seasonality is detected to be 12 units (1 year).
The confidence interval slider displays a shaded area that indicates the number of forecasted values that fall within a specified number of standard deviations. If there is a need to have a very high correlation for forecasts, select one standard deviation. This will also be an indication of how well the forecast model fits the data. The nature and requirements of the business problem and the user will determine an acceptable value for the confidence interval. For this data, 68% of expected values fall within one standard deviation.
There is also the ability to perform a hindcast. A hindcast produces a model that uses historical data to predict future values based on a preceding selected point in time. New predictions are generated that show how the current predictions would look if the prediction was generation at some past point in time.
Prior to this point, the appropriate model would have been selected (time series) and the model applied and evaluated. Within Power BI, the option to select a specific time series model is not available. With regards to model evaluation, adjustment of the confidence interval and hindcasting provides the ability to evaluate the overall fitness of the model.
Finally, the model is deployed and can be used for revaluation. This can be done via exporting the model along with its data to Excel and running it back through the forecasting model again.
It has been demonstrated how Power BI forecasting can be leveraged using the CRISPtm methodology and how advanced analytics can be placed in the hands of the masses. Power BI as a solution is simple to understand, uses existing technologies and is straightforward to implement. Over time, more and more advanced analytic capabilities will be exposed to the masses and to be successful, a well defined process, approach and appropriate training must be used to ensure that proper results and insights are obtained.
Questions and comments can be addressed directly to:
Director, Data Management – Strategist
Continuing the simplification of mobile first, cloud first from the previous post…
Let’s highlight the two big objectives that are achieved by separating core business services and platforms specific client in the last post:
- Platform and device outreach – HTTP being understood by all modern devices, makes your service consumable by any device that can host a client application and understand the language of the web.
- Heavy lifting done on the server – With the separation between a client app and business as a service running on a server somewhere, all the heavy lifting is done on the server where as the user’s device is doing mostly the user interaction. The heavy lifting work is generally referred to complex computations that consume a lot of hardware resources like CPU, RAM which is generally a limitation on small mobile devices.
Now let’s talk a little about the server.
Is your application business ready or feature ready?
So now we have built our application in a restful manner to reach a broad spectrum of devices and we moved the heavy lifting on the server. At this point our business idea can either take off or send us back to the drawing board. In either case the load on the server that is doing the complex operation is going to fluctuate.
The question here is – is the application infrastructure elastic enough to support that… or is such increase and decrease in the infrastructure going to come with a heavy cost?
It is a difficult question to answer for any developer – how many users (or traffic) would the current server infrastructure be able to hold? The best answer that you will get would be a very careful calculation based on perhaps stress testing, overly padded with seasoned wisdom. In fact, in case of a new application or a rewritten application with the newer frameworks, it is very difficult to evaluate the ideal infrastructure requirement until the rubber hits the road. To be on the safer side, every team tends to overestimate.
Cloudy with heavy awesomeness
Moving the infrastructure to cloud will help you achieve such elasticity. You do not need to worry about contacting data centers really, you can spin off new servers and shut them down when not needed, using a few lines of scripts. Depending on the service you are using, you could do many infrastructure operations using a self-service portal and be charged for only the infrastructure you use, for the duration you used it.
Suppose after we launched our application, we found that out target customers are in a specific geographic location like the east coast or some other part of the world that our analysts never imagined. Can you quickly respond to the new found opportunity? Most cloud service providers will allow you to select the geographic location of your infrastructure, allowing to place more servers closer to the customer for optimized user experience.
Global cloud providers are large organizations that have heavily invested in the infrastructure over the years thus providing you high security and availability. Therefor, there are many benefits that your business gets by moving to the cloud that might be difficult to estimate beforehand.
The device and service situation
Over the last decade, the tech industry has seen the exponential growth of a variety of devices (laptops, mobile phones, tablets, gaming consoles sensitive to touch/voice/motion/gestures). This is only going to get more diversified, whether we talk about 10 new devices within the 5.75 inch and 6.85 inches sector or the consoles / wrist bands that will replace medical equipment or watches that will talk to phones, glasses and TVs. Yes, the forecast is intense.
Relax, take some REST.
Given we can only partially foresee which future devices will be available, how do we maintain consistent delivery of business functionality to the devices that are still to be developed? Well, build a service that can communicate with all the existing devices and can also take care of the REST.
For decades we were happy with XML and SOAP messages communicating across applications. But now this communication has grown beyond traditional applications and devices. From some remote server hosted somewhere in the cloud to smart touchscreens, from set-top boxes to gaming consoles, some of those devices understand SOAP but many do not. However all of them positively understand HTTP. HTTP is what connects everything to the “Cloud” and staying away from either might not be a good idea.
Although the term “cloud” is heavily misused for sales purposes, I will come back to it a little later, let’s first talk about…
What is Service First?
So once you have identified that your business functionality will be delivered via HTTP, you should build your logic and let it be consumed via the HTTP service. HTTP service is more commonly known as RESTful service or REST service Web APIs. This is an architectural pattern that embraces HTTP as transportation protocol. So RESTful services are basically services built to be consumed over the web via HTTP.
Once we have functionality ready and available to be consumed across the spectrum, we can go ahead and create client apps for as many platforms and devices as we want. As far as maintenance is concerned, any change in the business logic will be one change to your service, and a consistent UX will represent the brand.