Vendors, especially in the over crowded security space, often must help buyers justify their investment. But what happens when there isn’t a real problem during the test period? This can make it difficult to properly assess. Some security vendors will simulate problems, others may sponsor penetration tests, or they may provide a list of tests and tools, and so on. In the highly competitive End Point market (aka AntiVirus) they will use any tool they may have in the box.
This article lists the problems around testing and I am *only* using this as an example, not judging. To balance this, I also see this in other industries. For example, in the automobile industry, when German manufacturer VolksWagen was involved in a scandal related to environmental emissions test manipulation.
I didn’t expect this topic to be so important, but the more I have spoken with customers I have come to realize that it is important to discuss tips for buyers to help them evaluate software solutions.
The process has many names - Proof of Value (POV), Proof of Concept/Capability (POC), Pilot, etc. And semantically they may have slightly different meanings but the key concept is allowing a customer to be able to test and evaluate a product’s capabilities prior to purchasing. For simplicity, I will refer to this as a POC.
Let’s take a closer look. As a software buyer, I want the evaluation process to answer a question:
Does it solve current (& future) problems and address the need?
Wearing my vendor hat, I want prospects to measure the product and find a fit between the solution and the organization.
Some enterprises conduct POCs in a limited, controlled environment, and others in actual authentic settings. It is also important to be looking at the bigger picture. Beyond the dry technicalities, there needs to be strong “product fit” the day after the POC is over. There is a difference between a product functioning in a controlled environment vs one that needs to bring value daily under tangible conditions.
A product can be tuned to fit POC or demo scenarios, and it is the customer's responsibility to verify that it functions well under real life scenarios, i.e. scale and ever changing conditions.
Here are some of the things you should be aware of when trying out a new offering - excluding the obvious conditions such as false positive and false negatives.
- Be careful of shiny objects!
What may be showing promise at the start, could stay as such. You must consider if this contributes to your needs beyond the WOW effect. For example, an analytics tool that is very powerful but requires a dedicated analyst, is a service not a product. You need to make sure that it meets expectations in terms of total cost of ownership - for example ensuring it is simple/easy to use vs requiring more dedicated time/resources to manage.
- Paid POCs “may” be a red flag
Paid POCs may be presented as a sign of commitment, but most likely hide complexity. Business stakeholders should plan tests to ensure products can be self sufficient the day the sponsored engineer sets foot out the door.
- Controlled POCs are wonderful….for the vendor
Sometimes it is impossible to test a solution in production. Perhaps this is because of internal politics or simply because it may affect the broader team and requires training before enrolling everyone. No matter the reason, you must evaluate and extrapolate the costs of operations: will the team be able to handle the solution? e.g. outputs and alerts - are they valuable or just nice to have?
- What happens in POC doesn’t stay in POC
Issues that are raised in POC may be a good sign of what to expect in real life. The team’s responsiveness, the scale, and all the little details which make a difference - test them thoroughly and have a clear commitment for fixes in case procurement happens.
- Aligned expectations help in POC
They say great leaders don’t just set expectations, but align them. What are your success criterias? Work with your vendor to list them, bridge gaps, foresee problems, map roles and responsibilities so you stay a leader. Vendor supplied success criterias will make the product look good, but may miss your business requirements.
- The future looks bright
What does the company’s future look like? When I examine a products for my company I check the vendor’s plans, execution rate and their roadmap. This helps me to estimate their intentions and to make sure the investment will not only bring value now, but also in the foreseeable future.
To sum up, POCs are tools that help an enterprise decide on product fit, estimate production costs, assess performance and overall working model delivery. With that said, it comes with responsibility and a need for transparency on both sides - the vendor and enterprise evaluating the solution. There is a set of activities that need to be completed before, during and after the POC. These steps should raise your chances of a more successful outcome and help you better align security with business objectives. Essentially this is what security leaders aspire to!