Sitemap Submit - Magento Extension to Submit your Google Sitemap

Sitemap Submit – Magento Extension to Submit your Google Sitemap

Posted by Ashley on Jan 30, 2010 • (21)

I’ve made a new Magento extension for submitting your store sitemap to Google, Yahoo, Ask and Bing. My new extension can submit automatically whenever the sitemap is regenerated, or on-demand from the admin UI. More about how I made it and how it works below, but first a little background.

How Sitemap Submit came to be

As of last week I have been taking a train to work in a far off land called Pukekohe a couple of times per week. Pukekohe itself is not traditionally known as a high-tech startup hub – but in 2010 the number of small high-tech software engineering companies has grown considerably (probably at least 200%).

What all this means is I get a couple of hours a day, two times a week where I’m literally stuck in front of my laptop as the train I’m on rocks and weaves it’s way across country to WWA HQ. This new Magento extension I have made is the first of hopefully many what I’ll now call Public-Transport-Projects. The extension will manually or automatically submit your Google sitemap file from your Magento store to several major search engines (Google, Yahoo, Bing and Ask).

How to use Sitemap Submit

To start with, you’ll need to install it – head over to the Sitemap Submit extension page on Magento connect to get started with that.

Once installed, It’s dead easy to use – basically you generate your sitemaps the usual way using Magento’s built in functionality – to do this go to the Catalog -> Google Sitemap section of the admin interface as shown in the screenshot below.

Sitemap Submit found in the Magento Sitemap admin menu option

Once you have your sitemap.xml generated you can click on it and view the details. One this edit screen you will see the new Submit button, which when clicked, my extension will submit your sitemap to Google, Yahoo, Ask and Bing. You can see in the screenshot below where the button should be.

The Sitemap Submit submit button on the sitemap edit screen of Magento Admin

There is also a configuration option for automatic submission (In System->Config->Catalog->Google Sitemap). What this means is that whenever your sitemap is generated or regenerated (by you or by cron if you have it configured to do so) then the sitemap will also be submitted to the search engines.

The Sitemap Submit Configuration options - an option to enable automatic submission each time the sitemap is generated, and also the yahoo App Id required for yahoo submission. Find them in System->Config->Catalog->Google Sitemap

The only other configuration to be aware of is the yahoo app id, to use the api you need to add an app id – read all about the Yahoo API and the app id. If you omit an ID, the extension will just skip notifying Yahoo, no big deal.

How Sitemap Submit works

I won’t bore you with too many details, basically it’s made up of an observer that watches out for the regeneration and a service object that actually does it (plus the usual various Magento extension things, like a controller and a helper.

The only tricky thing was the core Sitemap.php object didn’t fire a generating event, which if you read my explanation of Magento events and observers you’ll know is quite important. I override and slightly alter the default Magento sitemap as shown below:

 public function generateAction() {
    	parent::generateAction();
    	Mage::dispatchEvent('sitemap_sitemap_generate', array('sitemap'=>$this));
    }

That fires the event which is then picked up by my observer and in turn my submission service kicks off. It basically runs through each search engine and submits the sitemap using the required URL’s like so :

         	//ping google
        	if($this->_pingGoogle($url)) {
        		$msg = $msg . "Pinged Google successfully<br/>";
        	} else {
        		$msg = $msg . "Failed to ping Google.<br/>";
        	}

… and then I actually make the request like this:

 
	protected function _makeRequest($ping) {
 
		$curl = new Varien_Http_Adapter_Curl();
        $curl->setConfig(array(
            'timeout'   => 20
        ));
 
        $curl->write(Zend_Http_Client::GET, $ping, '1.1');
        $data = $curl->read();
 
        if ($data === false) {
            return false;
        }
 
        if ($curl->getInfo(CURLINFO_HTTP_CODE) == 200) {
	        // Mage::log($data); // uncomment to debug raw submission response
	        return $data;
        } else {
	        Mage::log("Submission to: " . $ping . " failed, HTTP response code was not 200");
	        return false;
        }
 
        //TODO: handle timeout?
 
	}

And that’s about it, you can read the full story in the open-source PHP code, in fact I’d invite you to do so, and give me some feedback on it!

The Sitemap Submit Feature Roadmap

I’ll be looking to extend the extension further over the next month or so in two ways. One is to add many more search engines that have an open sitemap submit function. The second is I want to enable more SEO capability and allow store submission to the many Magento store directories and Magento showcases like Magento Parade. I’m also, as always, really keen to hear feedback on fixes and features that you would like to see in the product.

21 Comments

Chris says:
February 3rd, 2010 at 6:41 am

When signing up for the Yahoo API what url should be used for the application entrypoint? Just the root website or is something more specific required?
Scott says:
February 18th, 2010 at 9:15 am

Would be nice if you could also build in sitemap splitting support for websites with more than Google’s 50K URL / 10MB limit.

Like this:
http://www.offshoresoftwaredevelopmentindia.com/blog/2009/06/13/magento-extension-google-xml-sitemap-split-module/

Not sure if these two extensions would work together or not.
Paul says:
February 20th, 2010 at 10:55 am

Not able to get 0.2.1 to show up in admin. (?)
Ashley says:
February 20th, 2010 at 12:57 pm

Can you confirm the files are installed in app/code/community/Aschroder? Also you have cleared your cache?
Paul says:
February 23rd, 2010 at 10:24 pm

Got it…Cache. Thanks
jerry says:
March 8th, 2010 at 4:38 am

How to work in multi websites ? generate it , always main site url.
Ashley says:
March 8th, 2010 at 6:08 pm

Good question – thanks to Filipe my extension will submit the correct Magento store URL to Google, Yahoo, Bing etc however it looks like a display bug in the Magento Core renderer (Mage_Adminhtml_Block_Sitemap_Grid_Renderer_Link) that causes it to use this:

$this->htmlEscape(Mage::getBaseUrl('web') . $fileName);

when it should probably use:

Mage::app()->getStore($sitemap->getStoreId()) ->getBaseUrl(Mage_Core_Model_Store::URL_TYPE_LINK) . $fileName;

The htmlescape() I’m not sure about – can any URL characters actually mess with HTML? I guess there could be spaces etc in the url path, perhaps I should escape my url too – anyone confirm this?
Ken says:
April 7th, 2010 at 11:17 am

Hi Ashley,
I’m having a minor issue with your Sitemap extension and was hoping you could help. When the sitemap is submitted to Google, Yahoo, etc. it’s being submitted as DOMAIN/index.php/sitemap.xml when the actual address is just DOMAIN/sitemap.xml – is there a simple way I can remove the “index.php” bit so that the various search engines can fine my sitemap rather than a 404 error? Thanks!!
Ashley says:
April 7th, 2010 at 1:20 pm

Hi, Please check the base url in your system configuration settings – perhaps the index.php is coming from there? Let me know.
Ken says:
April 7th, 2010 at 1:25 pm

No, nothing in there. I know I’m stuck with the index.php for most of my URLs until I can get the Base Rewrite to work correctly, but that’s not the case with the sitemap. Anyway, I’ll fiddle around and figure it out eventually, just thought you might have an idea. Thanks!
Ashley says:
April 7th, 2010 at 1:35 pm

oh – it might be a bug then – If url rewrites are off then I need to return the base URL without index.php, I’ll look into it when I get a moment. Thanks for the heads up.
Ken says:
April 7th, 2010 at 1:39 pm

Anytime. Keep up the great work!
Ken says:
April 12th, 2010 at 2:27 pm

Finally managed to get my rewrites working properly, so the sitemap extension is behaving just as it should for me now. Good luck w/ the potential bug fix & thanks again for a great addon!
Gary says:
April 13th, 2010 at 3:46 am

Hi Ashley,

I have tried the plugin on Magento 1.2.x, but cant see the set-up forms etc. Does it work on this platform?

Thanks,
Gary
Gary says:
April 13th, 2010 at 4:28 am

I should have mentioned that the specific issue is that I can enter the Yahoo API code etc under the configuration part, but that when I run the Submit option for the sitemap, I get the following error.

Fatal error: Call to undefined method Varien_Http_Adapter_Curl::getInfo() in /home1/myweb/public_html/mywebsite/app/code/community/Aschroder/SitemapSubmit/Model/Submit.php on line 126

However, manual generation does work as expected.

Gary
Ken says:
May 8th, 2010 at 12:43 pm

Hi Ashley,

I’ve just recovered from having my site hacked (grr, PHP code injection) & am concentrating on hardening security wherever I can. I noticed last night that the sitemap.xml file generated by your extension always sets itself to 666 permissions, meaning that it’s world writable. It’s easy enough for me to change the perms, but a bit of a headache to have to remember to redo it every time it gets generated. Is there something in the code I could change to make the file always generate with 644 permissions?

Thanks!!
- Ken
Ashley says:
May 8th, 2010 at 2:56 pm

Sorry to hear about your hack – was it a vulnerability in Magento or some other PHP script? God suggestion to limit the sitemap permissions.The sitemap.xml is actually generated by the built in Magento code, I just took a look though and I think I could add something to my extension to tweak the permissions. I’ll add it to the list of features for the extension, and update it when I’m next on the train.
Ken says:
May 8th, 2010 at 7:02 pm

Hi Ashley,

Thanks It was a nightmare to cleanup, but I think we’re more solid than ever now, so I ’spose it was for the best. It was actually vulnerabilities in an old WordPress installation I hadn’t updated or removed & wasn’t using – laziness on my part pure and simple.

I’ll keep my eyes here for updates & will just alter perms manually in the meantime.

Thanks again!
- Ken
jlapitan says:
May 29th, 2010 at 1:08 am

i cant find the configuration option for automatic submission
Ashley says:
May 29th, 2010 at 2:27 pm

It’s in System->Config->Catalog->Google Sitemap – I should make that clear in the blog post, thanks for the heads up.
Mike Reid says:
August 4th, 2010 at 5:14 am

Hi
I just installed this on 1.4.1.1 and it’s not showing up in the admin.
Is it not compatible with that version?