Hi DM,

For my message reader, when reading a sub-board, currently I have it using MsgBase.get_all_msg_headers() and then filtering it, removing vote headers and deleted messages (if the user can't view them), so it only displays messages the user can read.

It seems this is the best way to do that from JS (and it looks like msglist.js is basically doing the same thing?). Another sysop mentioned he has sub-boards with 100,000+ messages, and it can be fairly slow to load & filter them. I'm curious if there is a faster way to do that (or, is it possible for something faster to be made available to JS to do this task)?

Nightfox

---
■ Synchronet ■ Digital Distortion: digitaldistortionbbs.com

Re: Filtering a list of messages
By: Nightfox to Digital Man on Thu Nov 09 2023 01:17 pm

Hi DM,

For my message reader, when reading a sub-board, currently I have it using MsgBase.get_all_msg_headers() and then filtering it, removing vote headers and deleted messages (if the user can't view them), so it only displays messages the user can read.

You shouldn't have to remove vote messages manually if you pass the includes_votes argument as false.

It seems this is the best way to do that from JS (and it looks like msglist.js is basically doing the same thing?). Another sysop mentioned he has sub-boards with 100,000+ messages, and it can be fairly slow to load & filter them. I'm curious if there is a faster way to do that (or, is it possible for something faster to be made available to JS to do this task)?

MsgBase.get_all_msg_headers() was written (added to the MsgBase class) specifically for performance - it's the fastest way to read *all* the message header's in a message base in one operation. Subsequent filtering (e.g. of deleted messages), should be very fast, unless you're doing something unexpect (re-reading the message header or index of messages again?). I'd have to examine your code to for sure, but one thing this other sysop could do is compare the performance of msglist.js against your own message lister and see if the performanced is different or the same.

One optimization possible for loading *all* messages would be to load the message base *index* only (which is smaller/faster) and then only load message headers when needed (possibly doing this asynchronously, in the background, while displaying the first loaded message headers to the user). This does mean that searching/sorting might be limited until all the headers are loaded.

Another big optimization would be for *new* message scans or *your* message scans - in those scenarios, you wouldn't need or want to load all the messages headers of the msgbase since most of the time you'll only be displaying a small fraction of them (if any). There's plenty of room for optimization of large message bases during *scans*.
--
digital man (rob)

Breaking Bad quote #43:
Congratulations! You're now officially the cute one of the group.
Norco, CA WX: 74.7°F, 12.0% humidity, 6 mph ENE wind, 0.00 inches rain/24hrs

Re: Filtering a list of messages
By: Digital Man to Nightfox on Thu Nov 09 2023 03:15 pm

MsgBase.get_all_msg_headers() and then filtering it, removing vote headers

You shouldn't have to remove vote messages manually if you pass the includes_votes argument as false.

MsgBase.get_all_msg_headers() was written (added to the MsgBase class) specifically for performance - it's the fastest way to read *all* the message header's in a message base in one operation. Subsequent filtering (e.g. of deleted messages), should be very fast, unless you're doing something unexpect (re-reading the message header or index of messages again?). I'd have to examine your code to for sure, but one thing this other sysop could do is compare the performance of msglist.js against your own message lister and see if the performanced is different or the same.

One optimization possible for loading *all* messages would be to load the message base *index* only (which is smaller/faster) and then only load message headers when needed (possibly doing this asynchronously, in the background, while displaying the first loaded message headers to the user). This does mean that searching/sorting might be limited until all the headers are loaded.

Another big optimization would be for *new* message scans or *your* message scans - in those scenarios, you wouldn't need or want to load all the messages headers of the msgbase since most of the time you'll only be displaying a small fraction of them (if any). There's plenty of room for optimization of large message bases during *scans*.

Thanks - I think those are some good tips.

Nightfox

---
■ Synchronet ■ Digital Distortion: digitaldistortionbbs.com