Server crashing when indexing

Submitted by Jeff on Wed, 2006-07-12 10:49.Troubleshooting

I have a MySQL database with 3 around million records, which I use to create my dbsight index.

Once a week I have to completely re-create my dbsight index, because the data changes so much, many records will be deleted or amended, and there will be a lot of new ones.

The problem I have is that when I try to create the index it regularly uses so many server resources it causes resin and Apache to crash, and the indexing job is never completed.
I've tried changing the settings as much as I can to reduce the stress on the server, by experimenting with different values in the Advanced Settings screen, but nothing has been completely successful.

The only way I've found so far to stop the server from crashing is to disable Apache while the indexing process takes place. This reduces the load on MySQL and resin, so the indexing process can complete.
I don't really like doing this though as it means the sites on the server are unavailable while this happens - so I do it in the early hours of the morning and often have to stay up until 5 or 6 in the morning while I wait for it to finish, so I can re-enable Apache before going to bed.

I've come to the conclusion that the best way to solve the problem would be if there was some way to slow down the index creation process, reducing the load on the server. I don't mind leaving it running all night unsupervised if I know it's not going to crash.

Is there a way to do this by altering some of the existing settings, or is it something that can be added in the future?

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Submitted by will on Wed, 2006-07-12 15:59.

I totally understand your situation. I wonder increase resin's memory may be able to solve the problem. But if you are stuck with 32bit jvm, you can only increase memory to 2G, which may not be enough for your data.

You have a great suggestion! How about we add one option to disable searching when full-indexing? I guess it is not needed in incremental mode.

We should be able to add it quite easily. I will update this topic as soon as it's added to the beta release.

Submitted by will on Sat, 2006-07-22 17:22.

added on 1.2.9 beta as of today