A couple of thoughts on a phenomenon such as content discovery. Once again I came across such a problem in one of the projects and decided to write about it. The post will be useless to those who are aware of how a security audit is conducted. However, for me, as a software developer, this once became a revelation. In information security, content discovery refers to the technique of searching for hidden resources, mainly in web applications.
I first encountered this phenomenon several years ago, when friends contacted me and said that their clients began to disappear. It was a small company. They had a sales department and a small one-page website where potential customers would leave their phone numbers. Then the sales managers called back to the customer and made deals. So, from a certain point on, customers who came to the site were processed by competitors before the manager had even a chance to call them.
At first, employees were the main suspects. Then I decided to take a look at the source code of the site. It turned out that the URL /clients (I don’t remember exactly) without any authentication was available to anyone for a few hours, showing the list of customers, who submitted the request. This URL was being twitched by the CRM and the new customers were being added to its database.
At that time, it wasn’t clear to me how to locate this path on the site, because it did not appear anywhere – neither in sitemap.xml nor in robots.txt or anywhere else. I did not know anything about content discovery, so I gave it a thought and eventually dismissed it (the problem was already solved anyway). As it turned out, there are ways to search for such hidden resources. Let’s see what you can hide and how to find it.