htdig on mandriva 2008.0 "host not found"

This is driving me crazy:

I've installed htdig, htdig-web and apache on 2008.0 [Mandriva seperates
the cgi stuff into a seperate -web rpm].

I've configured apache to allow indexing of directories and run htdig
with the following result:

# su - htdig -c "/usr/bin/rundig -a -vvv"
ht://dig Start Time: Sun Feb 24 13:55:34 2008
1:1:http://myserver/
New server: myserver, 80
- Persistent connections: enabled
- HEAD before GET: enabled
- Timeout: 30
- Connection space: 0
- Max Documents: -1
- TCP retries: 1
- TCP wait time: 5
- Accept-Language:
Trying to retrieve robots.txt file
Trying local files
tried local file /var/www/html/robots.txt
pushed
pick: myserver, # servers = 1
> myserver supports HTTP persistent connections (infinite)
0:2:0:http://myserver/: Trying local files
tried local file /var/www/html/index.html
Local retrieval failed, trying HTTP
host not found
pick: myserver, # servers = 1
> myserver supports HTTP persistent connections (infinite)
ht://dig End Time: Sun Feb 24 13:55:34 2008
Deleted, not found: ID: 2 URL: http://myserver/
#

The first part of this I take to mean that the server has actually been
found, but later I get "host not found". I've tried:

# su - htdig -c "host myserver"
myserver.mydomain has address 192.168.1.28

and have also changed start_url in /etc/htdig/htdig.conf to localhost,
127.0.0.1 and 192.168.1.28 and still get the above result. I've also
tried commenting out local_urls and local_user_urls, to force htdig to
use http - no change, except for missing out the attempt at using the
local filesystem.

Apache's logs show no attempts at connection.

Any ideas?

Sarah

On Sun, 24 Feb 2008 09:11:00 -0500, Sarah <(E-Mail Removed)> wrote:

> # su - htdig -c "host myserver"
> myserver.mydomain has address 192.168.1.28
>
> and have also changed start_url in /etc/htdig/htdig.conf to localhost,
> 127.0.0.1 and 192.168.1.28 and still get the above result. I've also

Have you tried using myserver.mydomain in htdig.conf? I've just done
some limited testing, and it appears you cannot specify an ip address
in htdig.conf, and must specify a fully qualified name, that can be
resolved by dns lookup.

Regards, Dave Hodgins

--
Change nomail.afraid.org to ody.ca to reply by email.
(nomail.afraid.org has been set up specifically for
use in usenet. Feel free to use it yourself.)

David W. Hodgins wrote:
> On Sun, 24 Feb 2008 09:11:00 -0500, Sarah <(E-Mail Removed)> wrote:
>
>> # su - htdig -c "host myserver"
>> myserver.mydomain has address 192.168.1.28
>>
>> and have also changed start_url in /etc/htdig/htdig.conf to localhost,
>> 127.0.0.1 and 192.168.1.28 and still get the above result. I've also
>
> Have you tried using myserver.mydomain in htdig.conf? I've just done
> some limited testing, and it appears you cannot specify an ip address
> in htdig.conf, and must specify a fully qualified name, that can be
> resolved by dns lookup.
>
> Regards, Dave Hodgins

I've tried it now - no better :-(

I really don't understand it - how can it have problems finding the host
even when I've given it the IP address!!! And if "host myserver" works
in the same environment then I can only assume that htdig isn't using
the standard libraries for resolving the hostname.

Thanks for the suggestion.

Sarah

Sarah wrote:
> This is driving me crazy:

[Snip htdig failing utterly to index my site!]

> Any ideas?
>
> Sarah

Looking through Freshmeat I have found two other apps that have stock
Mandriva rpms. Both have been updated within the last year or so.

Hyperestraier: <http://hyperestraier.sourceforge.net/>

Swish-e: <http://swish-e.org/>

Anyone with experience of these?

Sarah

On Sun, 24 Feb 2008 18:08:43 -0500, Sarah <(E-Mail Removed)> wrote:

> I really don't understand it - how can it have problems finding the host
> even when I've given it the IP address!!! And if "host myserver" works
> in the same environment then I can only assume that htdig isn't using
> the standard libraries for resolving the hostname.

htdig and htsearch are old, unmaintained packages. I looked through them
quite a bit, some time ago, when debugging the creation of kde helpcenter
indexes, which also uses htdig and htsearch.

Not working with an ip address makes sense, if it's doing a lookup of
get ip from hostname. I don't remember exactly what it does, and no
longer have the source installed, to look at.

Why it's failing when the host command is working, I don't know. I do
know there are several methods for resolving the ip of a hostname.

For example, I'm running a local nameserver, with hodgins.homeip.net
defined as 127.0.0.1. The nameserver on 127.0.0.1 is specified as
the first server to search, by listing it in /etc/resovconf/resolv.conf/head.
The dynamically assigned ip is also updated at dyndns.org.

How I lookup the address will determine the result I get.

# dig +trace hodgins.homeip.net
hodgins.homeip.net. 60 IN A 216.240.2.97
;; Received 232 bytes from 203.62.195.75#53(ns5.dyndns.org) in 428 ms
# host hodgins.homeip.net
hodgins.homeip.net has address 127.0.0.1

I also have an ad sever, "mserv.zap2it.com", only in my /etc/hosts file,
directed to 127.0.0.1

The host command ignores the /etc/hosts file.
# host mserv.zap2it.com
mserv.zap2it.com has address 206.18.98.140
yet traceroute, and my web browser use it.
# traceroute mserv.zap2it.com
traceroute to mserv.zap2it.com (127.0.0.1), 30 hops max, 38 byte packets
1 hodgins (127.0.0.1) 0.116 ms 0.088 ms 0.051 ms

How are you getting myserver.mydomain resolved to an ip address?

Regards, Dave Hodgins

--
Change nomail.afraid.org to ody.ca to reply by email.
(nomail.afraid.org has been set up specifically for
use in usenet. Feel free to use it yourself.)

On Mon, 25 Feb 2008 13:21:22 -0500, Sarah <(E-Mail Removed)> wrote:

> Looking through Freshmeat I have found two other apps that have stock
> Mandriva rpms. Both have been updated within the last year or so.
> Hyperestraier: <http://hyperestraier.sourceforge.net/>
> Swish-e: <http://swish-e.org/>
> Anyone with experience of these?

I've installed both. They both require some work to get the indexing
configured correctly.

In both cases, I have some problems with the search results, that I
haven't sorted out yet.

With Hyperestraier, using http://hodgins.homeip.net/cgi-bin/estseek.cgi
to search for the word "Works", the resulting file names, linked to by
the search results are in the form file:///var/www/html/mimetex.html,
which of course only works if you are searching, from the localhost, and
most browsers will give you security warnings, when you try to access
the local file.

With swish, using http://hodgins.homeip.net/cgi-bin/swish.cgi to search
for the word "Works", the resulting file names, linked to by the search
results are in the form http://hodgins.homeip.net/var/www/html/mimetex.html,
which doesn't work, as the /var/www/html should not be included.

Both packages are designed to build indexes that can be searched from
the command line, as well as via cgi scripts. As above, this does cause
some problems, since the proper paths to the files, varies depending on
how you are accessing them. The correct link in the above examples
shoule be http://hodgins.homeip.net/mimetex.html.

I expect, If I dug through the documentation, these can both be fixed,
but it isn't obvious how, from a quick skim of the documentation.

I'd be inclined to sort out the dns problems with htdig/search, as once
that's solved, the both "just work", and have for a long time.

Regards, Dave Hodgins

--
Change nomail.afraid.org to ody.ca to reply by email.
(nomail.afraid.org has been set up specifically for
use in usenet. Feel free to use it yourself.)

David W. Hodgins wrote:

[some excellent stuff on his experiences with swish-e and hyperestraier]

> I'd be inclined to sort out the dns problems with htdig/search, as once
> that's solved, the both "just work", and have for a long time.
>
> Regards, Dave Hodgins

I'm going to press on with swish-e for a while - I get an impression
that it might be a bit easier than hyperestraier. In the long run if
htdig isn't maintained it will just get more troublesome. I'll report
back if I get any of them working.

Thanks again,

Sarah

David W. Hodgins wrote:
> On Sun, 24 Feb 2008 18:08:43 -0500, Sarah <(E-Mail Removed)> wrote:
>
>> I really don't understand it - how can it have problems finding the host
>> even when I've given it the IP address!!! And if "host myserver" works
>> in the same environment then I can only assume that htdig isn't using
>> the standard libraries for resolving the hostname.

> htdig and htsearch are old, unmaintained packages.

I've noticed. It even appears to need a patch for glibc >= 2.2.2, which
came out on 2001-02-16!

> How I lookup the address will determine the result I get.

Now that is useful info.

> How are you getting myserver.mydomain resolved to an ip address?

I'm using dnsmasq (thanks Simon Kelley, great package!) on another PC.
It crosses my mind that this might be the problem, since this is
presumably an unusual setup and Google has found no other similar reports.

> Regards, Dave Hodgins

Thanks very much for your input Dave. To be honest, the best option
seems to be to find a better (or at least better maintained) indexing
package. htdig seems to be abandonware. Odd, really, that it seems to be
better known than the alternatives.

Sarah

Sarah wrote:
> David W. Hodgins wrote:
>
> [some excellent stuff on his experiences with swish-e and hyperestraier]
>
>> I'd be inclined to sort out the dns problems with htdig/search, as once
>> that's solved, the both "just work", and have for a long time.
>>
>> Regards, Dave Hodgins
>
> I'm going to press on with swish-e for a while - I get an impression
> that it might be a bit easier than hyperestraier. In the long run if
> htdig isn't maintained it will just get more troublesome. I'll report
> back if I get any of them working.
>
> Thanks again,
>
> Sarah

I've managed to get swish-e to work. The key to generating the correct
urls in the search results is

(1) Building the index with the following in the configuration file:

ReplaceRules regex "|^/var/www/html/||"

(2) Insert in the swish.cgi configuration file:

prepend_path => 'http://mywebserver/'

Alternatively you can do the same with just:

ReplaceRules regex "|^/var/www/html/|http://mywebserver/|"


Sarah

On Tue, 26 Feb 2008 17:50:31 -0500, Sarah <(E-Mail Removed)> wrote:

> David W. Hodgins wrote:
>> How are you getting myserver.mydomain resolved to an ip address?
> I'm using dnsmasq (thanks Simon Kelley, great package!) on another PC.
> It crosses my mind that this might be the problem, since this is
> presumably an unusual setup and Google has found no other similar reports.

I've just noticed that mdns4_minimal has been added to hosts line in
/etc/nsswitch.conf and it appears to be intended to lookup names on
windows domains, or via the avahi daemon.

For anyone else, like me, who doesn't have a network, or doesn't use
windows for domain resolution, remove it.

Given the age of the htdig software, that is probably the problem with
it failing to resolv the hostname.

I'd set up a nameserver (aka bind), to resolve the hostname, rather
then use anything using m$ methods. I expect that would solve the
problem with htdig, if you choose to try it again.

Regards, Dave Hodgins

--
Change nomail.afraid.org to ody.ca to reply by email.
(nomail.afraid.org has been set up specifically for
use in usenet. Feel free to use it yourself.)