1) A popular issue and one that is particularly important at this time of the crawl/index build is what Ive come to call Google Ghosts.
Every so often a new site is put up on the internet, or a new page, and it appears in the index before the monthly update. Quite often it has good rankings at that point in time. But horrifyingly they often disappear again a few days or weeks later.
These ghostly appearances are often confusing to newbies, and sometimes to the experts! However, the explanation is quite simple.
Google has two types of crawler, its main crawler and the fresh crawler. The main crawler is capable of discovering new pages. If a new page is discovered immediately after a new index has been built then that pages discovery is of little use if it has to wait a month for a new index to be built.
The building of a new index also contains all the calculations of the non-query dependant factors involved in calculating a pages rank (for example, PageRank).
To fully utilise the pages, without waiting for the next set of index calculations Google must do something very simple. Guess its ranking. By guessing the new content is at least available to the searchers.
As it is a guess, the following things hold true:
1. The ranking of a ghost entry will not be likely to be the ranking of the page after the index.
2. The ghost entry must be removed from the database before the monthly creation of a new database. This removal will be temporary.
If you have such an entry in the index, then your major objective should be to get it on the list of pages on Googles fresh-list. To do this update the content of the page at regular, preferably daily, intervals.
Why do you want to be on Googles fresh-list? Because fresh-listed pages are crawled more frequently and updated in the index. If you are optimizing your on the page factors then being on the fresh-list is the difference between waiting a few days to be able to analyse the effectiveness of your changes and waiting for a month.
Originally posted by philwiley
Five days ago I put up a new mini site, and added it to Google via addurl.html rather than linking to it from one of my other sites and waiting for it to be found.
Three days later it was in the top 10 for all my important keywords, and got about 130 visitors.
Then after just one day it vanished. It didnt just drop from the top 10 positions, its now not listed in Google at all.
Theres nothing wrong with the site. No spam, no hidden links, no duplicate content, etc. And the site has had no downtime.
All I can think of is that its got something to do with having no incoming links...oh it does have a pop up if that matters.
Id rather not give you the url because readers of my book and ezine come here and Ive found that if I give urls some build almost duplicate sites.
Any suggestions why it might have vanished?
We get asked this question quite frequently and therefore we have an article prepared to answer this question for you.
What Happens To A NEW Webpage After Google Initially Spiders It?
A NEW Webpage is not included in Googles "Main Index" until ...
1.) It has been crawled by the "Main Crawl" spider
... And ...
2.) It has been through an update after it has been crawled by the "Main Crawl" spider.
Until both of those occurrences have happened, any rankings that a new Webpage achieves is not likely to be the actual rankings that it will achieve once it has settled into the "Main Index."
Google does two types of crawls ...
1.) The "Main Crawl"
... And ...
2.) The "Fresh Crawl"
A new Webpage is first crawled by the "Fresh Crawl" spider. (The exception to this would be the time period soon after Googles monthly update, as it could then be first crawled by the "Main Crawl" spider. Monthly updates usually start around the 20th - 28th day of the month and can last for several days.)
To distinguish the difference between the two spiders look at the range of their IP addresses.
1.) The "Main Crawl" Spider = 216.239.46.*
... And ...
2.) The "Fresh Crawl" Spider = 64.68.82.*
To explain what happens with a new Webpage, well assume that it is first crawled by the "Fresh Crawl" spider.
At some point in-between Googles monthly updates the "Fresh Crawl" spider crawls the new Webpage. (The new Webpage may have been initially found by a link during the "Main Crawl" or the link to the new Webpage may have been found by the "Fresh Crawl" but either way, it "reads" the new page with the "Fresh Crawl" spider.)
Quickly, the new page is evaluated, and is included in the search results, even though it hasnt been involved in an update and isnt included in Googles "Main Index." These rankings are then very vulnerable and unstable and change very frequently.
Next, the regularly scheduled monthly update occurs, which is also known as "The Google Dance" and this page does take part at some stage of the dance, but remember that it has not yet been "read" by the "Main Crawl" spider, so it does not end up in the "Main Index" yet.
When the monthly update is over, the new Webpage will still be considered a "Fresh" page.
Soon the "Main Crawl" spider will "read" this new Webpage, but it will have to wait for the next monthly update before it will take its place in Googles "Main Index."
Until then, Google wont display its Backward Links (inbound links), and its rankings will remain very vulnerable and unstable and apt to change quite often.
Now to put all of this into perspective.
If a new Webpage is first crawled by the "Fresh Crawl" spider and immediately AFTER the "Main Crawl" spider, it will go through two monthly updates and have to wait almost two months before its included in the "Main Index" and for its rankings to become stable and accurately positioned.
During this time period the new Webpage may come and go in the Google SERP (Search Engine Results Pages). Erratic behavior is the norm for such pages.
On the other hand, if a new Webpage is first crawled by the "Main Crawl" spider (usually the latter part of the month) it will only have to wait about a month to eventually get into the "Main Index."
When Web designers and site owners are not aware of how Google actually processes their NEW Webpages, they can become very confused about what is actually happening when they see their pages appear and disappear or rank high one minute and then not be found in the top 100, the next.
Happy SEOing And Well See You At The Top!
Sharon & Roy