USER AGENT User Agent: lub User Agent: * lub User agent: Googlebot Description: The user-agent syntax defines the direction in which indexing robots move – if necessary, of course. This can be done in two ways. If you want to report all bots, add “*” (asterisk). User agent: * – this way of writing says about it: “Every robot should follow in this direction.” If you want to report something to a specific robot, for example Googlebot, the notation looks like this User-agent: Googlebot – this line says “These instructions apply only to Googlebot”. DISALLOW The ‘Disallow’ statement is use to prevent bots from accessing specific folders or files.
Google to index images on your page
This means that if you don’t want, for example,, you put them all in a folder and exclude it. How-to? Let’s say you’ve already move all your photos to a Lithuania Email List folder calle “pictures”. Now you nee to tell the bots not to visit this folder for indexing User-Agent: * Disable: /photos The above two lines of text in the robots.txt file will keep the robots out of the photos folder. Note: If you forgot the “/” character after the Disallow statement, like this User-Agent: Disable: then the crawler will visit your site.
This specific instruction
Look at the first line and then read the second (ie “Disallow:”). What will happen? After that, the robot will feel like a fish in the water, because you KHB Directory banne him nothing. So it starts indexing all the pages and files. ALLOW Only a few indexing bots understand, one of which is, for example, Googlebot. Allow: The ‘Allow’ statement allows the bot to determine whether it can see a file in a folder locke with the ‘Disallow’ command. To illustrate this, let’s look at the previous example.