Every day the open internet publishes a huge amount of information that companies twenty years ago would have paid spies for. Competitors' prices, stock levels in their warehouses, new items in their range, customer reviews, job postings (which show where a competitor plans to grow), sale listings, tenders, mentions of Your brand. All of it is in plain sight, free, and updated every hour. The only problem is that a human is physically unable to read it all. A program is able to.
Parsing is the automatic collection of information from open sources. Syntactic analysis is the next step: turning the collected raw text into structured data You can work with. Together they give a business what I call "peripheral vision": the ability to see the market around You constantly, not in fits and starts whenever a marketer finds time for a competitor analysis.
Collecting is not enough, You need to understand
Many people think parsing is just downloading pages. Downloading really is easy, it is taught in the first year of university. The difficulty starts after that. One store writes the price as "1 299 UAH", another as "1299.00", a third shows two prices, old and new, a fourth hides the price until You log in, and a fifth calls the very same product something completely different and puts it in another category. Sites change their layout without warning, and yesterday's scraper brings back nothing today.
This is where syntactic analysis comes in. It is like the work of a translator who receives a stack of letters in ten dialects and has to bring them together into one table in the standard language. You need to recognise where the price is, where the name is, where the attributes are, match the same product across different sellers, throw away the garbage and notice when a source has broken. Without this part, parsing gives You not data but a dump. You cannot work with a dump, You can only store it, and that is expensive.
Where it turns into money
I have seen many parsing projects, and the ones that pay off are those where the data feeds straight into a decision someone makes. Here are a few typical examples.
- Dynamic pricing. An online store knows its main competitors' prices for key items and automatically adjusts its own within set limits. You do not need to be the cheapest everywhere, it is enough not to be the most expensive where the buyer compares.
- Lead generation. Purchase requests, tenders, new companies in registries, job postings showing that a company is growing and will soon need Your product. The system collects, filters and hands the manager an already sorted list.
- Mention monitoring. Reviews of Your brand and competitors on marketplaces, in the media, on forums. Negativity is noticed within an hour, not a month later when it has already become "common opinion".
- Assortment analysis. Which products competitors have added, what they are constantly out of (meaning demand exceeds supply), which categories are growing.
Notice that in every case there is an action at the end: change a price, make a call, reply to a review, buy stock. If there is no action, parsing turns into a pretty dashboard people look at for the first two weeks.
A client once asked us to collect "everything possible" about his market. We honestly did: dozens of sources, hundreds of thousands of items, beautiful charts. A month later he admitted he had opened the report twice. So we sat down and threw away ninety percent of what we had collected, leaving one table: items where the competitor is cheaper by more than a set threshold, and items the competitor has out of stock. His purchasing manager opened that table every morning. Less data, more money. That is how it almost always goes.
Honest limitations
I would be dishonest if I did not mention the risks. First, legal. Collecting open data is generally legal, but sites have terms of use, personal data is protected, and there is a difference between collecting prices and collecting someone else's customer base. At Ingello we always start a project with a conversation about what may be collected and what may not, and sometimes that conversation ends the project. Second, ethical and technical: You must not flood someone else's site with thousands of requests per second. A good scraper behaves like a polite visitor, not like a crowd at a clearance sale. Third, maintenance: sources change constantly, and a parser without monitoring quietly dies within six months while You keep making decisions on stale data.
A parser without monitoring is like a sentry who has fallen asleep but is still standing at his post. From the outside everything looks fine.
How to start
If the idea sounds interesting, do not start by choosing a tool. Start with one question: which single decision in Your business would You make better if You knew more about the market? Prices for a hundred key products? A list of companies that need Your service right now? Buyers' reaction to a competitor's new release? Pick one, find five to ten sources where that information is openly available, and run a one-month pilot. Measure whether the decision changed and what it was worth in money.
At our company collecting and analysing open data has long been part of everyday work: our own client search and market monitoring systems are built this way, and we bring the same approach into client projects, connecting the collected data with the CRM and warehouse in Forma so it lands where decisions are made. The information around You is already there. The only question is who reads it first. Attentive robots to You, colleagues.